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1.0  EXECUTIVE  SUMMARY 

1.1  BACKGROUND 

The  U.S.  Army  Corps  of  Engineers  (USACE)-Huntsville  estimates  that  approximately  20 
million  aeres  of  land  are  eontaminated  with  unexploded  ordnanee  (UXO)  or  ordnanee  explosive 
waste  (OEW)  within  the  eontinental  Ehited  States.  This  is  the  result  of  operations  that  inelude 
training,  testing,  manufaeture,  storage,  disposal,  and  intentional  burials.  Now,  through  either 
eongressional  mandates  sueh  as  the  formerly  used  defense  sites  (EUDS),  base  realignment  and 
closure  (BRAC),  or  state  litigation,  these  sites  are  being  evaluated,  remediated  (as  required),  and 
certified  as  safe  for  the  intended  ultimate  public  use.  This  is  a  very  slow  and  costly  process,  and 
in  some  cases,  impossible  to  do  with  existing  technologies  and  resources. 

Within  the  Environmental  Security  Technology  Certification  Program  (ESTCP),  the  goal  of  the 
UXO  Cleanup  Thrust  is  to  develop  automated,  economical,  and  efficient  methods  to  accomplish 
UXO  site  cleanups  through  the  demonstration  and  fielding  of  new  technologies.  One  of  these 
funded  programs  was  the  Multi- Sensor  Towed  Array  Detection  System  (MTADS)  developed  by 
the  Naval  Research  Eaboratory  (NRE  )  [1].  This  system  consists  of  a  tow  vehicle  and  two  low 
self- signature  tow  platforms,  one  for  an  eight- sensor  magnetometer  array  and  the  other  for  a 
three-sensor  time  domain  electromagnetic  (EM)  pulsed  induction  array.  MTADS  uses  the  global 
positioning  system  (GPS)  for  navigation,  sensor  position  location,  and  survey  guidance  and  has  a 
sophisticated  data  analysis  system  for  interpreting  field  data.  Target  analyses  include  target 
position,  depth,  orientation,  predicted  target  size,  and  a  goodness- of- fit  value  [2].  Based  on  the 
success  of  extensive  field  demonstrations,  NRE  signed  a  Cooperative  Research  and  Development 
Agreement  (CRADA)  with  a  commercial  company,  Blackhawk  Geometries,  Inc.,  and  non¬ 
government  systems  have  been  designed  and  built  based  on  the  NRE  vehicular  and  man- portable 
MTADS.  These  systems  are  currently  in  commercial  use  providing  UXO  srvices  to  the 
Department  of  Defense  (DoD)  [3]. 

The  USAGE  has  estimated  that  vehicle  towed- array  survey  systems  can  be  used  effectively  on 
only  about  30%  of  UXO -contaminated  land  because  of  rugged  terrain,  buildings  on  or  near  the 
site  to  be  surveyed,  or  wooded  sites.  GPS  navigation  systems  do  not  operate  effectively  without 
a  clear  view  of  the  sky,  which  limits  use  of  systems  like  MTADS  under  forest  canopies.  ESTCP 
funded  NRE  to  develop  an  MTADS  man- portable  magnetometer  system  (MMS)  in  EY  98  and  an 
EM  man-portable  system  (EMMS)  in  EY  99.  Each  platform  was  implemented  with  both  GPS 
and  acoustic  navigation  systems  to  allow  surveying  in  areas  without  sky  view.  The  system 
hardware  designs  allowed  MMS  and  EMMS  survey  data  to  be  incorporated  with  vehicular 
survey  data.  This  required  implementation  of  a  new  data  acquisition  system  for  both  the 
vehicular  and  the  man-portable  systems  and  modification  of  the  data  analysis  system  to 
incorporate  and  seamlessly  overlay  all  data  sets  [4]. 

The  Navy  Tri-Service  Environmental  Quality  Research  Development  Test  and  Evaluation 
Strategic  Plan  specifically  addresses,  under  Thrust  Requirements  l.A.l  and  1.A.2,  the 
requirements  for  improved  detection,  location,  and  removal  of  UXO  on  land  and  under  water. 
The  requirements  document  states: 


1 


There  are  more  than  20  million  acres  of  bombing  and  target  ranges  under  DOD 
control.  Of  particular  concern  for  the  Navy  are  the  many  underwater  sites  which 
have  yet  to  be  characterized.  Each  year  a  significant  fraction  (200,000-500,000 
acres)  of  these  spaces  are  returned  to  civilian  (Private  or  Commercial)  use.  All 
these  areas  must  be  surveyed  for  buried  ordnance  and  other  hazardous  materials, 
rendered  safe  and  certified  for  the  intended  end  use.  This  is  an  extremely  labor 
intensive  and  expensive  process,  with  costs  often  far  exceeding  the  value  of  the 
land....  Improved  technologies  for  locating,  identifying  and  marking  ordnance 
items  must  be  developed  to  address  all  types  of  terrain,  such  as  open  fields, 
wooded  areas,  rugged  inaccessible  areas,  and  underwater  sites  [5]. 

The  MTADS  addresses  all  aspects  of  the  Tri-Service  Requirements  for  land-based  buried  UXO. 
It  is  capable  of  detecting  all  classes  of  buried  UXO  at  their  maximum  likely  penetration  depths. 
The  system  correctly  locates  buried  targets,  determines  their  burial  depths,  classifies  the  likely 
ordnance  size,  and  provides  for  future  target  way  pointing,  as  well  as  creating  geographical 
information  system  (GlS)-compatible  target  output  maps  and  sorted  target  tables.  The  man- 
portable  adjuncts  extend  this  capability  into  areas  of  rugged  terrain  and  areas  with  poor  sky 
visibility  [4]. 

1.2  L-RANGE  DEMONSTRATION 
1.2.1  L-Range  Demonstration  Objectives 

In  August  1999,  the  man-portable  systems  were  evaluated  in  a  demonstration  at  the  L-Range  at 
Blossom  Point,  Maryland. 

The  great  strengths  of  the  MTADS  are  its  sensitivity,  which  allows  detection  of  all  ordnance  to 
its  maximum  self-burial  depths;  the  position  location  accuracy  of  the  navigation  and  positioning 
system;  the  target  analysis  algorithms  that  allow  location  of  buried  objects  to  within  the  actual 
ordnance  volume;  and  the  analysis  output  products  that  provide  for  the  efficient  reacquisition  and 
remediation  of  the  targets.  A  design  goal  for  the  man- portable  projects  was  to  preserve  the 
detection  sensitivity  for  buried  ordnance  while  extending  the  survey  capability  to  include  rugged 
terrain  and  open  forest  areas. 

The  performance  objectives  for  the  L-Range  demonstration  were  to: 

•  Conduct  MMS  and  EMMS  surveys  of  the  entire  survey  area  using  the  acoustic  navigation 
system; 

•  Conduct  MMS  and  EMMS  surveys  of  the  accessible  areas  using  the  GPS  navigation 
system; 

•  Serially  conduct  MTADS  target  analyses  for  all  man-portable  surveys  and  the  vehicular 
surveys  —  using  the  3-B  classifier  for  analyzing  the  vehicular  EM  survey  data; 
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•  Develop  dig  lists  based  on  each  analysis  and  prepare  dig  images  to  reacquire  and  flag  all 
targets  picked  from  each  survey; 

•  Dig  all  targets  using  UXO- qualified  technicians — document  location,  orientation,  and 
identification  of  all  recovered  targets  and  photograph  all  UXO  in  place  before  removal; 
and 

•  Analyze  results  to  determine  the  relative  performance  achieved  using  each  survey 
approach. 

1.2.2  L-Range  Demonstration  Results 

The  MMS  and  the  EMMS  were  demonstrated  on  the  L-Range  at  the  Army  Research  Laboratory 

during  the  week  of  September  13,  1999.  Part  of  the  site  was  in  the  open  near  a  tree  line,  the 

remainder  in  the  woods  where  the  acoustic  system  was  exclusively  used  for  navigation.  In  open 

areas,  duplicate  surveys  were  conducted  with  each  navigation  system. 

The  following  is  a  summary  of  the  key  results  from  the  L-Range  demonstration. 

•  The  MMS  system  uses  the  same  sensors  as  the  MTADS  vehicular  system.  Data  from  the 
two  systems  are  effectively  equivalent,  interchangeable,  and  indistinguishable. 

•  The  man-portable  EM  survey  equipment  is  much  less  sensitive  than  the  vehicular  array. 
Depending  on  target  size  and  depth,  the  0.5  m  X  1.0  m  coil  produces  peak  signals  12  to 
16  times  smaller  than  that  of  the  vehicular  array. 

•  Coverage  in  open  areas  was  relatively  good.  Coverage  in  wooded  areas  was 
unsatisfactory  with  relatively  large  areas  unsurveyed.  The  difficulty  of  using  the  EMMS 
in  tight  areas,  combined  with  excessive  backpack  loads,  difficulty  following  lane  layouts, 
poor  site  survey  management,  and  operator  frustration  all  contributed  to  poor 
performance. 

•  The  MMS  detected  all  the  targets  that  were  characterized  using  the  vehicular 
magnetometer  array.  The  vehicular  magnetometer  survey  and  the  MMS/GPS  survey 
results  demonstrated  that  the  MMS  can  provide  equivalent  field  performance  to  the 
vehicular  system. 

•  In  the  EMMS  survey  using  GPS  navigation,  14  targets  were  missed  that  were 
characterized  in  the  vehicular  EM  survey  and  in  all  the  magnetometer  surveys.  In  every 
case,  a  target  was  missed  because  either  there  was  no  measurable  signal  in  the  EM  data, 
or  the  signal- to- noise  ratio  placed  the  target  below  the  limit  required  for  analysis.  The 
navigation  error  caused  by  the  rocking  of  the  GPS  antenna  created  an  additional  noise 
source  that  tended  to  smear  out  a  target  signal  and  make  it  very  ragged. 
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•  The  acoustic  navigation  system  transmitter  does  not  significantly  interfere  with  the  EM 
sensor.  However,  use  of  the  acoustic  navigation  system  degrades  the  location  accuracy 
relative  to  the  GPS  system. 

Following  the  L-Range  demonstration,  it  was  concluded  that  although  the  adjuncts  met  most  of 
the  design  performance  specifications,  they  were  unsuitable  in  their  present  configurations  for 
use  as  commercial  field  instruments.  Numerous  upgrades  and  modifications  were  made  during 
FY  2000,  the  third  year  of  the  program,  in  preparation  for  the  final  demonstration  of  both  adjunct 
systems  at  the  U.S.  Army  Jefferson  Proving  Ground  (JPG)  in  southeast  Indiana  in  August  2000. 

1.3  JEFFERSON  PROVING  GROUND  DEMONSTRATIONS 

1.3.1  JPG-V  Demonstration  Objectives 

The  focus  of  the  JPG-V  demonstration  was  to  assess  the  capabilities  of  the  top  detection 
technologies  identified  during  previous  JPG  demonstrations  under  more  realistic  conditions.  The 
intent  was  to  quantify  each  system’s  detection,  discrimination,  cost,  and  production  rates  while 
operating  at  several  areas  within  JPG  containing  natural  (magnetic  rocks  and  soils)  and  man¬ 
made  (munitions  fragments)  clutter.  The  survey  test  areas  consisted  of  three  1- hectare  prepared 
sites  located  near  the  40- acre  site  used  during  the  JPG- IV  demonstrations  [6]. 

The  demonstration  approach  was  built  around  a  scenario  intended  to  evaluate  technologies  that 
might  address  the  particular  demands  of  the  Kaho’olawe  Island  cleanup.  Both  inert  UXO, 
ordnance  and  explosives  (OE)  scrap,  and  magnetic  soils  and  rocks  were  incorporated  into  the 
test.  Two  of  the  three  sites  took  advantage  of  naturally  occurring  magnetic  soil  deposits.  These 
were  augmented  by  soil  and  rock  samples  from  Hawaii.  The  third  site  was  relatively  benign 
geologically.  Each  demonstrator  was  required  to  conduct  digitally  mapped  georeferenced 
surveys  and  to  conduct  target  analyses  on  site  as  though  concurrent  remediation  would  take 
place.  Consequently,  each  demonstrator  was  responsible  for  determining  the  best  method  of 
employing  his  system  to:  (a)  ensure  full  coverage  of  each  test  area;  (b)  collect  high-quality 
sensor  data  to  support  detection  and  discrimination  requirements;  (c)  achieve  high  production 
rates;  and,  (d)  minimize  man- hour  requirements  and  costs. 

1.3.2  JPG-V  Demonstration  Results 

The  NRE  EMMS  demonstrated  the  highest  degree  of  maturity  and  preparation,  and  conducted 
the  field  surveys  and  onsite  analysis  with  no  problems.  Both  the  MMS  and  the  EMMS 
demonstrated  the  capability  to  suppress  high  magnetic  background  from  geologic  sources  and  to 
provide  high-quality  (well- localized,  high  signal- to-background  target  signatures)  georeferenced 
data.  The  EMMS  outperformed  the  mag- and- flag  approach  at  all  three  test  areas,  and  the  single¬ 
point  performance  points  met  the  Kaho’olawe  Tier  II  requirements.  However,  the  EMMS  did 
not  achieve  100%  detection  at  any  of  the  three  sites.  EMMS  mean  depth  estimation  errors  were 
well  within  the  0.5  m  allowable  error. 

In  terms  of  production  rates,  the  EMMS  was  the  best  performer  among  the  advanced  technology 
demonstrators,  achieving  an  average  of  1  hectare  per  6.56  hours  and  requiring  a  field  survey 
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crew  ranging  from  one  to  four  persons.  In  terms  of  costs,  the  EMMS  was  again  the  best 
performer  with  total  costs,  including  assessed  penalties,  ranging  from  11.4%  to  27.1%  lower  than 
the  other  demonstrators. 

1.4  THE  KAHO’OLAWE  DEMONSTRATION 

During  the  fall  of  2001,  ESTCP  sponsored  a  demonstration  of  hand-held  and  man- portable 
electromagnetic  induction  (EMI)  systems  on  prepared  sites  on  the  island  of  Kaho’olawe  [7].  The 
three  better  demonstrators  from  the  JPG-V  demonstrations  were  invited  to  participate  [8]  and 
GTE,  using  a  TM-5  EMU  frequency  domain  sensor  was  incorporated  into  the  demonstration. 
GTE  had  worked  on  the  main  island  cleanup  the  previous  year  and  the  site  managers  felt  the 
system  had  merit.  Parsons-UXB  employed  the  EM61  systems  used  on  the  main  island  cleanup. 
Eor  the  primary  surveys,  the  wheeled  EM61  were  used  in  an  EM-and-Elag  mode  using  pin  flags 
to  mark  targets.  During  the  ESTCP  demonstration,  the  pin  flag  positions  were  acquired  using 
hand-held  GPS  systems.  Parsons-UXB  used  the  standard  EM61  integrated  with  a  GPS  system  in 
a  gridded  survey  of  the  ESTCP  demonstration  areas,  the  system  also  used  in  this  mode  for 
quality  assurance  (QA)  studies  on  the  main  island  cleanup. 

The  demonstration  managers  prepared  a  calibration  survey  area,  seeded  with  ordnance  and  frag 
challenges.  Ground  truth  was  provided  to  each  demonstrator  before  they  occupied  the  site. 
Demonstrators  were  allowed  1  week  to  work  with  the  calibration  site  before  moving  on  the  main 
demonstration  sites  during  the  second  week. 

The  demonstration  site  was  adapted  from  the  QA  range,  which  had  previously  supported  the 
main  island  cleanup.  Additional  ordnance  and  clutter  targets  were  installed  prior  to  the  ESTCP 
demonstration  surveys.  The  demonstration  area  was  divided  into  three  contiguous  sites.  Area  A 
was  60m  X  60  m.  Area  B  was  60m  X  90  m  and  Area  C  was  90m  X  ~1 10  m  (1  hectare). 

1.4.1  The  Kaho’olawe  Demonstration  Objectives 

The  demonstration  objectives  were  effectively  identical  to  those  incorporated  into  the  JPG-V 
demonstration.  Each  demonstrator  was  required  to  conduct  digitally  mapped  georeferenced 
surveys  and  on-site  target  analyses  as  though  concurrent  remediation  would  take  place. 
Consequently,  each  demonstrator  was  responsible  for  determining  the  best  method  of  employing 
his  system  to  ensure  full  coverage  of  each  test  area,  collect  high-quality  sensor  data  to  support 
detection  and  discrimination  requirements,  achieve  high  production  rates,  and  minimize  man¬ 
hour  requirements  and  costs.  Deliverables  were  ranked  target  dig  lists  and  2- dimensional 
anomaly  survey  maps.  Each  demonstrator  was  required  to  provide  location,  depth,  and  size 
information  for  each  selected  anomaly;  to  attempt  to  identify  ordnance  by  type  for  ordnance 
declarations,  and  to  attempt  to  differentiate  between  ordnance  and  clutter. 

1.4.2  Kaho’olawe  Demonstration  Results 

Eollowing  analysis  of  the  data  from  the  calibration  site  during  week  one  of  the  demonstration, 
NRE  concluded  that  only  about  50%  of  the  UXO  targets  on  the  site  were  realistically  detectable. 
Sensor  noise,  metallic  clutter  noise,  and  geological  interferences  made  the  smallest  shallow 
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targets  and  the  deeply-buried  bombs  and  projectiles  undetectable.  The  enhanced  sensitivity  of 
the  NRL  EMI  system,  its  relocation  with  minimal  ground  clearance,  and  the  shock  sensitivity  of 
the  receive  coil  units  worked  as  disadvantages  for  the  EMMS  relative  to  the  other  EM61  units 
demonstrated  at  this  site. 

1.5  REGULATORY  DRIVERS 

There  are  no  regulatory  issues  unique  to  the  man- portable  MTADS  adjuncts.  The  primary 
regulatory  issue  affecting  UXO  detection  and  discrimination  technologies  is  gaining  confidence 
and  approval  from  federal,  state,  and  local  regulators,  stakeholders,  and  users.  Acceptance  of 
these  innovative  technologies  from  agencies  such  as  the  USAGE  and  the  Naval  Eacilities  and 
Engineering  Command  is  needed  to  ensure  that  future  requests  for  proposals  (REP)  for  UXO 
cleanup  projects  will  be  written  in  a  manner  that  will  either  sanction  these  technologies  or  at  least 
allow  their  inclusion  in  proposals  for  UXO  site  work. 
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2.0 


TECHNOLOGY  DESCRIPTION 


2.1  BACKGROUND  AND  APPLICATIONS 

The  MMS  and  the  EMMS  have  many  common  components.  The  primary  platform  for  each 
sensor,  a  wheeled  cart,  provides  relatively  smooth  operation,  permits  coordinated  deployment  of 
all  sensor  and  navigation  hardware  during  surveys,  and  maintains  the  sensors  at  a  fixed  height 
above  the  ground.  By  restricting  sensor  dimensions  and  the  array  design  to  the  width  of  a 
person’s  shoulders,  the  system  should  be  deployable  in  areas  where  a  person  could  easily  walk. 

The  initial  system  requirements  and  specifications  for  the  MMS  and  EMMS  are  summarized  in 
Table  1  [4].  Note  that  these  requirements  and  specifications  were  for  the  MMS  and  the  EMMS 
as  configured  for  the  E-Range  demonstrations. 

Table  1.  System  Requirements  and  Specifications  for  the  MTADS  Man-Portable 

Adjuncts. 


System  Specification/ 
Requirement 

MMS 

EMMS 

Continuous  operating  time 

2  hours 

2  hours 

Survey  area  (single  setup) 

1  acre 

1  acre 

Lane  spacing 

0.5  m 

0.5  or  1.0  m 

Sensor  sensitivity 

0.1  nT  (same  as  vehicular  MTADS) 

Scalable  to  vehicular  MTADS 

Sensor  data  rate 

10  Hz 

10  Hz 

Navigation  data  rate 

GPS  5  Hz 

Acoustic  1  Hz 

Sensor  position  accuracy 

GPS  0.1  m 

Acoustic  0.25  m 

Same  as  MMS 

Sensor  height  above  ground 

0.25  m  (fixed) 

0.25  m  (fixed) 

Data  acquisition  system  (DAQ) 

Compatible  with  vehicular  DAQ  based  on 
modified  Geometries  858  data  recorder 

Same  data  recorder  as  the  MMS 

Data  analysis  system  (DAS) 

Seamless  integration  with  vehicular  data 

Same  as  MMS 

2.2  COMPONENT  DESCRIPTIONS  [4] 

2.2.1  Sensors 

The  MMS  sensors  are  Geometries  858  ROV  Cesium  Vapor  magnetometers,  identical  to  those  on 
the  MTADS  vehicular  system.  Geonics,  Inc.  developed  new  EM  sensors  for  the  EMMS.  The 
design  of  the  0.5  m  X  1.0  m  coil  system  allows  the  system  to  be  used  as  0.5  mwide  instrument 
in  tight  or  confined  spaces  or  as  1.0  m-wide  system  in  more  open  applications.  The  sensitivity  of 
the  0.5  m  coil  was  supposed  to  be  increased  by  additional  turns  in  the  transmit  coil. 

2.2.2  Platforms 

Both  the  MMS  and  the  EMMS  are  designed  with  the  sensors  and  the  navigation  antennas 
mounted  on  two -wheeled  carts.  The  operator  wears  a  specially  designed  backpack  containing 
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batteries  and  navigation  hardware.  Figure  1 
shows  the  prototype  MMS  survey  system 
operating  with  the  GPS  navigation  system. 

The  commercial  Geonics  EM61  coils  are 
delivered  with  a  two -wheeled  transport  system 
(Figure  2).  Either  the  0.5  m  X  1.0  m  or  the 
1.0  m  X  1.0  m  coils  may  be  used  as  a  single 
cart.  The  cart  and  antenna  support  system 
rigidly  hold  the  top  and  bottom  coils  in  place, 
with  the  GPS  or  acoustic  antenna  center- 
mounted  above  the  upper  EM  coil.  Rigidity  is 
required  because  the  GPS  antenna  has  a 
significant  signal  at  the  upper  coil,  which  must 
be  nulled  as  part  of  data  preprocessing. 
Because  of  the  excessive  weight  of  the  system 
batteries  and  the  GPS  instrumentation,  the 
equipment  was  split  between  two  backpacks 
for  the  E- Range  demonstration  (see  Eigure  2). 
When  using  the  acoustic  navigation  system,  all 
equipment  fits  into  one  backpack  thus 
permitting  a  single  operator  to  take  survey 
data. 

2.2.3  Data  Acquisition  System  (DAQ) 

The  modified  Geometries  858  data  recorder 
shown  in  Eigure  3  is  the  backbone  of  the  man- 
portable  survey  equipment.  The  standard 
Model  858  commercial  system  is  configured 
with  dual  cesium  vapor  sensors  and  comes 
with  one  RS-232  COM  port  access  to  the 
palmtop  data  storage  device.  Two  battery 
packs  are  included  with  operating  times  of  6 
hours  each.  The  data  logger  stores  2.5  Mb  of 
data  or  about  4.5  hours  in  the  MMS 
configuration  in  which  the  GPS  is  recorded  at 
5  Hz  and  the  sensors  at  10  Hz. 

The  Model  858,  as  modified  for  use  with  the 
MMS  and  the  EMMS,  has  three  additional 
COM  ports  to  accommodate  GPS  GGK  string 
input  at  5  Hz,  acoustic  navigation  data  from 
the  Ultra  Sonic  Ranging  and  Data  System 
(USRADS)  Data  Pack  at  1  Hz,  and  either  1 


Figure  1.  MMS  Prototype  System  Deployed  at  the 
L-Range  Demonstration. 


Figure  2.  EMMS  Prototype  System  Deployed  at  the 
L-Range  Demonstration. 


Figure  3.  Geometries  Model  858  Data  Logger. 
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pulse  per  second  from  the  Trimble  7400  receiver  or  the  sync  pulse  from  the  acoustic  navigation 
system. 

The  Geometries  858  can  be  mounted  on  the  sensor  platform  as  shown  in  Figure  3  or  worn  in  a 
fanny  pack  on  the  operator’s  waist. 

2.2.4  Navigation 

Trimble  Model  7400  GPS  receivers,  currently  used  by  the  vehicular  MT ADS,  were  also  used  for 
the  MMS  demonstration  at  the  L-Range.  This  system,  operating  with  a  Model  4000  SSI  base 
station  receiver,  provides  the  full  range  of  GPS  location  options.  In  the  highest  precision  or 
level  3  fix  (real-time  kinematic  [RTK]),  the  base  station  provides  differential  position  fixes  with 
fully  resolved  numerical  ambiguities,  yielding  real-time  location  accuracies  of  2-5  cm.  A  level  2 
fix  (Float)  is  accurate  only  to  10-50  cm  because  some  of  the  integer  ambiguities  remain 
unresolved  in  the  solution.  The  level  4  fix,  the  traditional  differential  solution,  may  have  its 
accuracy  degraded  to  0.5- 1.0  m  depending  on  the  number  and  position  of  the  satellites  in  the 
solution.  MTADS  surveys  are  begun  with  only  a  level  3  fix.  During  a  survey,  if  the  fix  quality 
briefly  drops  to  level  2,  offsets  and  corrections  are  implemented  in  the  data  cleanup  process. 
Survey  data  based  on  lower  quality  fixes  are  generally  not  used  in  MTADS  surveys. 

The  Chemrad  Navigation  System  (USRADS  Model  2300)  provides  acoustic  navigation  data 
when  satellite  visibility  will  not  support  GPS  navigation.  This  system  includes  an  acoustic 
transmitter  on  the  rover  platform  and  a  network  of  up  to  10  transponders  at  fixed  stations 
deployed  about  the  perimeter  of  a  survey  site.  The  system  uses  time- of- flight  (speed- of- sound) 
data  to  triangulate  among  all  possible  pairs  of  transponders  to  create  a  location  position  which  is 
updated  at  10  Hz.  Based  on  evaluation  testing  in  wooded  areas,  the  USRADS  system  can 
provide  navigational  fixes  accurate  to  ~25  cm  when  10  stationary  receivers  are  used  to  enclose  a 
1-acre  wooded  site.  Under  good  conditions,  the  acoustic  sensors  have  a  range  of  about  200  ft. 
This  accuracy  is  degraded  in  either  high  wind  or  high  noise  environments,  or  when  visibility  is 
limited  by  obstructions.  Deployment  strategies  are  critical  to  the  successful  use  of  this  system. 

2.2.5  Data  Analysis  System  (DAS) 

The  MTADS  data  analysis  system  (DAS)  has  been  modified  to  accept  data  from  the  MMS,  to 
fully  integrate  vehicular  and  man- portable  data  or  multiple  MMS  data  sets,  and  to  provide  the 
same  analysis  capabilities.  When  preprocessing  is  complete,  magnetometry  data  from  either  the 
MMS  (using  either  GPS  or  acoustic  navigation  systems)  or  the  vehicular  system  can  be 
combined  into  single  survey  data  sets.  For  instance,  wooded  areas  within  a  larger  vehicular 
survey  area  would  be  surveyed  with  the  MMS  (and  acoustic  navigation)  and  edited  to  fill  in  areas 
inaccessible  to  the  vehicular  MTADS.  In  general,  data  from  multiple  surveys  of  the  same  areas 
are  not  commingled,  the  better  data  being  used  for  any  given  area. 

This  approach  is  not  used  for  data  taken  with  the  EMMS.  Because  of  the  different  sensitivities 
of  the  various  EM  coil  systems  and  the  various  deployment  options,  EMMS  data  are  processed 
separately  from  vehicular  EM  data.  The  baseline  MTADS  DAS  currently  has  all  necessary 
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utilities,  routines,  and  switches  to  allow  processing  of  data  from  all  vehicular  and  man-portable 
setups  regardless  of  the  navigation  system  used  to  survey. 

2.3  PREVIOUS  TESTING  OF  THE  TECHNOLOGY 

The  MTADS  vehicular  system,  on  which  the  MMS  and  EMMS  are  based,  was  extensively 
demonstrated  at  the  following  locations. 

•  NRL’s  Chesapeake  Bay  Detachment,  Chesapeake  Beach,  Maryland,  October  1996  [8]. 

•  Marine  Corps  Air  Ground  Combat  Center,  Twentynine  Palms,  California,  December 
1996  [9]. 

•  Jefferson  Proving  Grounds,  Madison,  Indiana,  January  1997[10]. 

•  Badlands  Bombing  Range  on  the  Pine  Ridge  Reservation,  South  Dakota,  July  1997  [11]. 

•  The  former  Fort  Pierce  Naval  Amphibious  Base,  Vero  Beach,  Florida,  January  1998  [12]. 

•  The  former  Buckley  Field,  Aurora,  Colorado,  June  1998  [13]. 

•  Faguna  Pueblo  Reservation  in  New  Mexico,  July  1998  [14]. 

•  Portsmouth  Naval  Shipyard,  Kittery,  Maine,  October  1998  [15]. 

•  The  JPGIV  Data  Fusion  Demonstration,  October  1998  [16]. 

•  Walker  River  Reservation,  Shurz,  Nevada,  November  1998  [17]. 

Based  on  the  success  of  these  field  demonstrations,  NRF  signed  a  CRADA  with  a  commercial 
company,  and  a  nongovernment  system  has  been  designed  and  built  based  on  the  NRF  MTADS. 
This  system,  including  both  the  vehicular  and  man- portable  platforms,  is  currently  in  commercial 
use. 

2.4  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

There  are  many  advantages  associated  with  these  systems.  The  MMS  and  EMMS  were  designed 
to  extend  the  MTADS  survey  capability  into  rugged  terrain  that  cannot  be  traversed  by  the  tow 
vehicle.  The  acoustic  navigation  system  extends  the  survey  capability  into  areas  with  limited  sky 
view,  including  those  under  forest  canopies.  The  wheeled  MMS  cart  is  designed  to  be  pushed 
ahead  of  the  operator,  allowing  the  operator  to  maneuver  through  fairly  tight  spaces.  The  width 
of  the  cart  and  sensor  arrays  are  designed  to  be  at  or  below  the  width  of  the  operator’s  shoulders. 
Wider  EM  coils  are  available  for  areas  with  less  severe  space  constrictions. 

One  disadvantage  of  this  design  is  that  access  is  precluded  by  areas  that  are  very  tight  and  by 
areas  that  have  difficult  ground  cover  such  as  brush,  deep  mud,  or  extremely  rugged  terrain. 
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3.0 


L-RANGE  DEMONSTRATION  DESIGN 


3.1  PERFORMANCE  OBJECTIVES 

The  primary  objectives  of  the  MMS  and  EMMS  projects  were  to  develop  and  demonstrate  man- 
portable  adjuncts  to  the  vehicular  MTADS  to  support  UXO  surveys  and  remediation  operations 
in  areas  that  are  too  rugged  for  operation  of  the  vehicular  MTADS  and  in  areas  without  adequate 
sky  view  for  effective  use  of  the  GPS  navigation  system  [18]. 

The  overall  performance  objectives  for  the  MMS  and  EMMS  adjuncts  to  the  vehicular  MTADS 
include  the  following. 

•  The  detection  sensitivity  of  the  adjunct  sensor  platforms  should  match  those  of  the 
vehicular  arrays  to  the  greatest  extent  possible. 

•  A  single  DAS  should  support  both  the  magnetometer  and  EM  platforms  operating  with 
either  GPS  and  acoustic  navigation. 

•  Acoustic  navigation  data  must  be  compatible  with  and  mergeable  with  GPS-based  data. 

•  The  man-portable  DAQ  output  formats  must  be  compatible  with  the  vehicular  MTADS 
DAS. 

•  Survey  data  from  the  man-portable  platforms  must  seamlessly  merge  with  vehicular  data. 

•  Performance  of  the  man-portable  systems  must  meet  the  requirements  and  specifications 
in  the  program  development  plan. 

3.2  PERFORMANCE  METRICS 

3.2.1  Target  Location  Accuracy 

The  test  field  surveys  were  conducted  with  each  of  the  vehicular  MTADS  platforms  and  with  the 
MMS  and  EMMS  using  both  GPS  and  acoustic  navigation.  The  man-portable  demonstration 
area  on  the  E-Range  was  seeded  with  18  inert  ordnance  items,  12  in  GPS- accessible  areas,  and 
six  in  the  woods.  The  positions  of  these  items  were  precisely  known.  Twelve  of  the  items 
appear  in  all  six  surveys.  The  six  items  in  the  woods  were  surveyed  only  with  the  man-portable 
systems.  Comparisons  among  the  systems  are  made  as  well  as  evaluations  of  the  absolute  target 
location  accuracies. 

3.2.2  Survey  Integration 

MMS  and  EMMS  surveys  from  the  wooded  areas  were  integrated  with  their  GPS  counterparts  on 
the  open  range  and  with  vehicular  MTADS  surveys.  Target  analyses  were  carried  out  on  these 
merged  data  sets  to  verify  that  the  DAS  software  modifications  accommodated  integration  of 
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man- portable  data  with  vehicular  data  and  integration  of  acoustic  navigation  data  with  GPS 
navigation  data. 

3.2.3  System  Operational  Performance 

Field  logs,  to  be  kept  for  all  survey  activities,  provided  information  about  setup  times,  survey 
times,  production  rates,  and  equipment  performance.  Specific  attention  was  paid  to  issues 
relating  to  battery  operational  times  in  the  field,  file  creation  and  data  storage  capabilities,  and 
the  performance  of  the  highly  modified  Geometries  Model  858  data  recorder.  Field  notes  were 
made  about  the  operational  performance  of  the  man-portable  carts  in  open  areas  and  in  the 
woods.  Experiences  with  the  setup  of  the  acoustic  sensors  in  open  areas  and  in  the  woods  were 
recorded  along  with  weather  conditions  at  the  time  of  the  surveys. 

The  electronic  data  files  provided  additional  information  about  field  survey  performance  by 
documenting  survey  data  collection  times,  the  course  and  paths  used  in  data  collection,  stoppages 
during  data  collection,  how  turnarounds  were  accomplished,  and  how  well  surveyors  lined  up  to 
begin  their  survey  paths.  Missed  areas  were  determined,  and  difficulties  with  navigational  data 
were  documented. 

3.2.4  Detection  Capability 

The  magnetometers  on  the  vehicular  and  man- portable  systems  are  identical,  as  are  their  spacing 
and  heights  above  ground.  Data  acquisition  rates  are  set  to  provide  similar  data  densities  from 
both  man- portable  and  vehicular  surveys.  Therefore,  differences  between  man- portable  and 
vehicular  surveys  should  be  a  result  of  platform  and  motion- induced  noise  levels  and  the  relative 
ability  of  the  navigation  systems  to  accurately  track  the  positions  of  the  sensors.  MTADS  DAS- 
generated  plots  of  individual  sensor  measurements  as  a  function  of  time  were  used  to  evaluate 
and  quantify  the  relative  importance  of  various  noise  sources  in  the  data  and  their  effect  on  the 
detection  sensitivity  of  the  man- portable  and  vehicular  magnetometry  systems. 

The  EMMS  sensor  records  a  significantly  smaller  signal  over  any  individual  target  than  does  the 
vehicular  EM  array.  The  absolute  detection  sensitivity  of  the  EMMS  system  was  determined 
from  test  pit  and  test  field  data.  Relative  signal- to- noise  issues  and  their  effect  on  detection 
sensitivity  were  evaluated  in  the  same  manner  as  described  in  the  previous  paragraph  for  the 
man- portable  and  vehicular  magnetometry  systems. 

3.3  SITE  SELECTION 

An  area  at  the  Army  Research  Eaboratory  Blossom  Point  facility  was  chosen  for  the  man- 
portable  demonstration  site.  A  part  of  the  E-Range  is  shown  in  an  aerial  photograph  in  Eigure  4. 
This  area,  800  feet  wide  by  5000  feet  long  and  encompassing  about  95  acres,  has  been  a  test 
range  for  various  munitions.  The  range  is  bordered  on  the  south  (river  side)  by  a  mixed  pine  and 
hardwood  forest.  The  range  itself  is  ^nerally  flat,  grass  covered,  and  regularly  mowed.  The 
forest  bordering  the  range  varies  from  open  woods  to  rugged  areas  with  extensive  underbrush. 
The  3- acre  area  designated  for  the  ESTCP  data  fusion  demonstration  is  outlined  in  yellow.  The 
MMS  and  EMMS  demonstration  areas  are  outlined  in  red.  The  entire  area  within  the  red 
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boundary  was  surveyed  with  the  EMMS  and  MMS  platforms  using  the  acoustic  navigation 
system.  The  portion  of  the  site  accessible  to  the  GPS  navigation  system  was  similarly  surveyed 
with  each  portable  system.  The  entire  area  within  the  yellow  boundary  was  surveyed  with  both 
the  magnetometer  and  EM  vehicular  MTADS  arrays. 


Figure  4.  L- Range  at  Blossom  Point. 


3.4  TEST  SITE/FACILITY  HISTORY  AND  CHARACTERISTICS 

The  Army  Research  Eaboratory’s  Blossom  Point  Test  Range  in  Charles  County,  Maryland,  on 
the  tip  of  Cedar  Point  Neck  near  Ea  Plata — approximately  50  miles  south  of  Washington,  D.C. — 
has  been  used  for  ordnance  testing  and  research  since  1942.  During  World  War  II,  Harry 
Diamond  and  his  team  at  the  National  Bureau  of  Standards  (NBS),  now  the  National  Institute  of 
Standards  and  Technology  (NIST),  needed  open  areas  where  they  could  test  fuzes.  In  early 
1943,  NBS  leased  land,  establishing  this  proving  ground  for  proximity  fuzes  at  Blossom  Point. 
By  September  1945,  14,000  rocket  and  mortar  rounds  had  been  fired  on  various  ranges  on  the 
site. 

In  1953,  the  lease  on  the  property  was  transferred  to  the  Army,  which  operated  the  property  as  a 
fast- reaction,  low-cost  range  for  experimental  work.  During  the  Vietnam  War,  the  Army’s  Harry 
Diamond  Eaboratory  was  very  active  at  the  site.  After  the  war,  the  Army  phased  out  Blossom 
Point,  intending  to  transfer  the  work  to  Aberdeen.  The  facility  was  reactivated  in  January  1976 
for  continued  research  on  fuzes.  In  1980,  the  Army  purchased  the  property  for  $2.7  million. 
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3.5  PHYSICAL  SETUP  AND  OPERATIONS 


The  entire  L- Range  was  surveyed  using  the  vehicular  MTADS  magnetometer  array.  Survey 
areas  were  selected  to  assure  that  many  buried  targets  would  exist  within  the  demonstration  area, 
but  that  the  density  of  buried  clutter  would  not  be  so  high  as  to  preclude  analysis  of  most  buried 
targets  as  single  items.  This  was  particularly  important  in  the  data  fusion  demonstration,  as 
analysis  algorithms  intended  to  recover  target  shape  information  were  being  evaluated.  The 
survey  areas  selected  could  be  extended  seamlessly  into  the  woods  and  included  both  open  and 
wooded  areas. 

UXO  benchmark  items  were  seeded  into 
the  demonstration  area  to  provide 
additional  ground  truth  for  evaluating  the 
relative  performance  of  the  MMS  and 
EMMS  using  each  navigation  system.  The 
dimensions  of  the  survey  area  were 
expanded  over  those  shown  in  Figure  4  (in 
red)  to  allow  placement  of  seed  targets  in 
relatively  clear  areas.  Figure  5  shows  a 
portion  of  the  vehicular  MTADS 
magnetometer  survey  with  the  expanded 
site  perimeter  outlined  in  white.  This 
image  provides  an  impression  of  the 
densities  of  buried  ferrous  targets  on  the 
F- Range.  The  man- portable  survey  area 
extends  an  additional  35  meters  into  the 
woods. 

The  inert  items  detailed  in  Table  2  range 
in  size  from  M-23s  (three- lb  practice 
bombs)  to  105-mm  projectiles.  Vehicular 
MTADS  surveys  were  used  to  choose  burial  sites  near  the  woods  that  were  relatively  clear  of 
other  buried  objects.  Inert  UXO  seed  targets  buried  in  the  woods  were  sited  using  hand-held 
metal  detectors  to  define  relatively  clear  burial  sites.  Backhoes  and  shovels  were  used  to 
excavate  for  the  seeded  items.  Items  buried  in  open  areas  were  located  using  GPS,  whereas 
items  buried  in  the  woods  were  located  using  laser  survey  equipment  operating  from  GPS 
benchmarks.  Orientations  and  depths  to  the  ordnance  centers  were  recorded  prior  to  backfilling 
and  tamping  of  the  soil  to  minimize  subsidence.  While  the  seed  items  were  being  buried,  the 
wooded  survey  site  was  partially  cleared  of  brambles  and  underbrush.  No  trees  or  large  saplings 
were  removed,  and  some  areas  were  left  with  brush  cover  that  would  be  difficult  to  walk  through 
with  the  MMS  and  EMMS.  It  was  expected  that  10-15%  of  the  wooded  site  would  not  be 
accessible  to  either  man-portable  instrument. 

Before  beginning  the  man-portable  surveys,  the  vehicular  MTADS  was  used  to  survey  the  area 
outlined  in  yellow  in  Figure  4,  again  using  both  the  magnetometer  and  EM  arrays.  The  survey 
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Figure  5.  Magnetic  Anomaly  Map  from  a  Vehicular 
MTADS  Survey. 
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was  extended  to  get  as  close  as  possible  to  the  woods  while  maintaining  acceptable  GPS 
navigation  fix  quality. 


Table  2.  Identification  and  Locations  of  the  Inert  UXO  Seed  Targets. 


Target 

No 

UXO  ID 

Location 

Location  Universal 
Transverse  Mercator  (UTM) 

Depth 

(m) 

Azi.* 

(deg) 

Inch** 

(deg) 

Longitude 

Latitude 

Northing 

Easting 

T-1 

81-mm 

38.408114616 

-77.103106356 

4253192.0 

316371.4 

0.21 

45 

45 

T-2 

M-23 

38.408182890 

-77.103135897 

4253199.7 

316369.0 

0.15 

0 

0 

T-3 

4.2-in 

38.408279866 

-77102816841 

4253209.8 

316.397.1 

0.63 

45 

45 

T-4 

105-mm 

38.408362672 

-77102673710 

4253218.7 

316409.8 

0.69 

45 

45 

T-5 

2.75-in 

38.408226790 

-77.102714295 

4253203.7 

316405.9 

0.30 

90 

0 

T-6 

M-23 

38.408258385 

-77.102623238 

4253207.0 

316413.9 

0.18 

45 

0 

T-7 

81-nini 

38.408306797 

-77.102575035 

4253212.3 

316418.2 

0.43 

90 

T-8 

81-nini 

38.408288164 

-77.102524670 

4253210.1 

316422.6 

0.28 

0 

0 

T-9 

2.75-in 

38.408321734 

-77.102495520 

4253213.8 

316425.2 

0.28 

0 

45 

T-10 

81-nini 

38.408324947 

-77.102430749 

4253214.0 

316430.9 

0.51 

45 

0 

T-11 

M-23 

38.408367373 

-77.102281185 

4253218.4 

316444.0 

0.20 

90 

T-12 

M-23 

38.408374777 

-77.102240512 

4253219.2 

316447.6 

0.15 

45 

45 

T-13 

2.75-in 

38.407993265 

-77.102011424 

4253176.4 

316466.7 

0.30 

0 

90 

T-14 

2.75-in 

38.408122587 

-77.102093780 

4253190.9 

316459.8 

0.30 

90 

T-15 

M-23 

38.408107204 

-77.102238338 

4253189.5 

316447.1 

0.15 

0 

0 

T-16 

M-23 

38.40993279 

-77.102292141 

4253176.9 

316442.2 

0.15 

45 

0 

T-17 

60-nini 

38.407977164 

-77.102425350 

4253175.4 

316430.5 

0.30 

0 

45 

T-18 

81-nini 

38.407903350 

-77.102445624 

4253167.3 

316428.5 

0.48 

45 

45 

*  Azi.  =  Orientation  in  the  plane  of  the  Earth’s  surface;  0  (deg)  is  the  vector  pointing  true  north. 

**  Inch  =  Target  dip  angle  relative  to  the  Earth’s  surface;  0  (deg)  is  in  the  surface  plane;  90  (deg)  refers  to  the  target  nose  pointing  down. 


The  corners  of  the  survey  areas  were  established  by  the  GPS  surveyor  and  marked  by  bicycle 
flags.  In  preparation  for  the  man- portable  surveys,  wooden  stakes  were  driven  at  1-m  intervals 
along  the  east  and  west  boundaries  of  the  survey  area.  Twine  was  used  to  connect  the  east  and 
west  stakes  defining  the  survey  lanes.  In  the  woods,  additional  stakes  were  used  within  the  site  to 
maintain  lane  spacings  as  the  twine  snaked  around  trees  and  through  the  brush. 

3.6  SAMPLING  AND  MONITORING  PROCEDURES 

MMS  and  EMMS  surveys  were  conducted  using  0.25- m  lane  spacings.  During  surveys  using  the 
GPS  navigation  system,  survey  times  were  chosen  to  allow  good  GPS  coverage  while  working 
adjacent  to  the  woods.  A  single  acoustic  navigation  beacon  setup  was  used  to  conduct  the  MMS 
and  EMMS  surveys  in  the  open  area  shown  in  Eigure  5.  A  second  acoustic  navigation  beacon 
setup  was  used  to  conduct  the  surveys  within  the  woods.  Man-portable  surveys  of  the  open  area 
required  3-3.5  hours  with  each  system.  Surveys  of  the  wooded  area  required  about  3  hours  each, 
including  setup  of  the  acoustic  navigation  system. 

As  shown  in  Table  3,  parts  of  the  man-portable  survey  area  were  covered  by  seven  separate 
surveys — two  vehicular  EM  surveys,  one  vehicular  magnetometer  survey,  two  MMS  surveys 
(with  GPS  and  acoustic  navigation),  and  two  EMMS  surveys  (GPS  and  acoustic  navigation). 
The  area  in  the  woods  was  surveyed  only  by  the  MMS  and  EMMS  using  the  acoustic  navigation 
system. 
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Table  3.  Survey  Log  for  the  Vehicular  and  Man- Portable  Surveys, 


Platform 

Navigation 

Survey  Area 

Survey  Date 

Survey  Time 

Vehicle/ mags 

GPS 

Eusion/MP  open  area 

7/28/99 

126  minutes 

Vehicle/EM  (East/West) 

GPS 

Eusion/MP  open  area 

8/3/99 

294  minutes 

Vehicle/EM  (North/South) 

GPS 

Eusion/MP  open  area 

8/3-4/99 

267  minutes 

MMS 

GPS 

MP  open  area 

8/5-6/99 

208  minutes 

EMMS 

GPS 

MP  open  area 

8/6/99 

196  minutes 

EMMS 

Acoustic 

MP  open  area 

MP  woods 

8/9-11/99 

216  minutes 

145  minutes 

MMS 

Acoustic 

MP  open  area 

MP  woods 

8/12/99 

8/10-11/99 

170  minutes 

165  minutes 
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4.0  L-RANGE  PERFORMANCE  ASSESSMENT 


4.1  PERFORMANCE  DATA 


In  the  man- portable  magnetic  120 

survey  shown  in  Figure  6,  the  area 
within  the  white  perimeter 
constitutes  the  man- portable  survey 
area.  The  area  south  of  the  red  line 
was  within  the  woods  and  could 
not  be  surveyed  using  the  GPS 
navigation  system.  The  largest 
missed  area  within  the  woods  was  ^ 
excluded  because  it  contained  a  ^ 

dead  deer.  Because  there  were  no 
seed  targets  within  that  area,  it  was 
not  surveyed.  Other  apparent 
missed  areas  included  groups  of  40 

trees  or  brush  that  could  not  be 
navigated  with  the  magnetometer 
cart.  Some  of  the  seeded  targets  20 

are  identified  in  the  image  and 
labeled  with  their  target  numbers. 


Figure  7  shows  two  of  the  EM 
surveys.  The  east- west  vehicular 
MTADS  survey  is  shown  on  the 
left,  whereas  the  EM  man-portable 
survey  using  acoustic  navigation  is 
shown  on  the  right.  Two  visual 
impressions  are  immediately 
apparent.  Eirst,  the  man-portable  EM  survey  equipment  is  much  less  sensitive  than  the  vehicular 
array.  Note  the  sensitivity  levels  on  the  plots.  Depending  on  target  size  and  depth,  the  0.5  m  X 
1.0  m  coil  produces  peak  signals  12-16  times  less  intense  than  that  of  the  vehicular  array.  The 
peak  signal  strength  from  the  1.0  m  X  1.0  m  man- portable  EM  coils  is  about  four  times  less 
intense  than  that  of  the  vehicular  array.  The  sensitivity  of  the  0.5  m  and  1.0  m  coils  scale 
directly  with  the  area  of  the  transmit  and  receive  coils.  Only  the  0.5  m  X  1.0  m  man- portable 
coils  were  used  in  this  demonstration.  The  results  of  this  decreased  sensitivity  becomes  apparent 
when  the  performance  of  the  system  is  analyzed. 
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Figure  6.  Magnetic  Anomaly  Image  of  the  Man-Portable 
Survey  Area  Taken  Using  Acoustic  Navigation. 


Second,  the  EMMS  survey  coverage  in  the  woods  was  poor  compared  with  the  coverage  of  the 
MMS  survey,  the  result  of  poor  quality  control  in  the  survey  process.  Either  a  complete  survey 
was  not  taken  or  survey  files  were  lost,  leaving  gaps  in  the  coverage.  The  extent  of  the  missed 
areas  was  not  realized  until  it  was  too  late  to  recover  and  resurvey  those  areas.  In  retrospect,  the 
wooded  areas  should  have  been  surveyed  in  orthogonal  directions  to  improve  coverage.  In  a  real 
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Figure  7.  EM  Anomaly  Images  of  the  Man-Portable  Survey  Area  Showing  the  MX  ADS  Vehicular  Survey 
Using  GPS  (left)  and  the  EMMS  Survey  Using  Acoustic  Navigation  (right). 


world  operation,  to  assure  complete  coverage  and  detection  of  all  UXO,  hand  surveys  using 
metal  detectors  would  be  required  to  survey  in  brushy  areas  and  in  missed  areas  adjacent  to  trees. 

A  summary  of  the  recovered  items  is  shown  in  Table  4.  Recovered  OE  scrap  items  included 
shrapnel,  tail  fin  assemblies,  and  spent  fuzes.  Nonordnance  related  materials  included  pieces  of 
iron  or  steel  such  as  angle  iron,  bolts,  hardware,  or  coils  of  wire  or  cable.  Some  ordnance  items 
recovered  were  split,  broken,  or  bent.  Several  were  challenged  with  1-in  shape  charges. 

Table  4.  Results  of  Target  Recovery  in  the  Man-Portable  Survey  Area. 


Item 

Seed 

Targets 

60-mm 

Mortar 

81 -mm 
Mortar 

4.2 -in 

Mortar 

25-lb  Frag 
Bomb 

5 -in 

Rocket 

OE 

Scrap 

Non-OE 

Scrap 

Number  recovered 

18 

1 

14 

1 

1 

1 

43 

11 

4.2  PERFORMANCE  CRITERIA 

Performance  evaluation  criteria  include  system  detection  sensitivity,  accuracy  of  location  and 
depth  predictions,  navigation  system  performance,  and  operational  and  ergonomic  efficiency  of 
the  man- portable  units. 
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4.3 


PERFORMANCE  ASSESSMENT 


4.3.1  Detection  Sensitivity 

The  ability  to  detect  targets  depends  primarily  on  the  sensitivity  of  the  sensors,  the  completeness 
of  the  survey  coverage,  and  the  signal-to-noise  ratio  in  the  survey  data.  The  MMS  system  uses 
the  same  sensors  as  the  MTADS  vehicular  system.  The  sensor’s  horizontal  separation  and  height 
above  the  ground  are  identical  to  the  vehicular  system,  and  data  sampling  rates  are  such  that  the 
sampling  density  of  the  two  systems  are  approximately  equivalent.  The  signal-to-noise  ratios  in 
data  taken  by  the  two  systems  are  also  similar.  Carefully  laid  out  surveys  conducted  with  the 
MMS  provide  similar  detection  and  performance  capabilities  to  the  vehicular  system.  Data  from 
the  two  systems  are  effectively  equivalent,  interchangeable,  and  indistinguishable. 

The  EMMS,  as  configured  for  the  LRange  demonstration,  was  deployed  with  0.5  m  X  1.0  m 
coils.  Based  on  the  areas  of  the  send  and  receive  coils  and  a  50%  overlap  of  transmitters  in  the 
vehicular  system,  one  would  expect  the  0.5  m  instrument  to  record  six  times  lower  signal 
strengths  than  the  vehicular  system  for  the  same  test  object.  Although  this  would  lower  the 
sensitivity  of  the  EMMS  relative  to  the  MTADS  array,  it  was  not  expected  to  significantly  limit 
its  ability  to  detect  targets  because  the  MTADS  array  system  is  extremely  sensitive  to  small, 
shallow  tarots.  In  reality,  the  new  coil  systems  were  not  completely  equivalent  to  the  units  in  the 
MTADS  array.  Measurements  with  electronic  test  equipment  showed  somewhat  different  system 
setup  parameters.  Measurements  on  a  test  stand  using  standard  objects  confirmed  the  lower 
sensitivity. 

In  addition,  as  deployed  at  the  E- Range,  the  EMMS  using  the  wheel  mounting  system  supplied 
with  the  instrument  placed  the  lower  coil  0.4  meters  above  the  ground.  The  vehicular  MTADS, 
depending  on  the  suspension  system  being  used  on  the  tow  vehicle,  typically  deploys  the  coils 
about  0.25  meters  above  the  ground.  Eigure  8  shows  a  comparison  of  a  small  part  of  the  E 
Range  survey  area  taken  from  the  vehicular  and  man-portable  surveys.  Note  the  difference  in 
display  scales.  The  EMMS,  as  used  in  this  demonstration,  is  10-15  times  less  sensitive  than  the 
vehicular  MTADS.  The  absolute  differences  are  also  a  function  of  the  size  and  depth  of  the  test 
objects.  The  EMMS,  as  configured  for  this  demonstration,  provides  data  sample  densities 
equivalent  to,  or  slightly  higher,  than  the  vehicular  system. 

The  deployment  of  the  GPS  antenna  high  above  the  coils  on  the  EMMS  introduces  an  additional 
noise  source  in  the  data.  When  the  system  traverses  over  rough  areas,  side- to- side  rocking 
motions  can  easily  displace  the  antenna  by  half  a  meter  horizontally  from  the  center  of  the  coils. 
Because  the  system  is  behind  the  operator,  this  motion  often  goes  unnoticed  or  uncontrolled. 

The  acoustic  navigation  system  transmitter  does  not  significantly  interfere  with  the  EM  sensor. 
The  transmitter  is  typically  deployed  at  the  center  of  the  upper  EMMS  coil. 


19 


Target  detection  capability  depends  on  being 
able  to  deploy  the  sensor.  To  test  the  ability  of 
the  man- portable  systems  under  realistic  survey 
conditions,  much  of  the  survey  area  was  in  the 
woods  and  much  of  that  area  was  left  relatively 
uncleared.  The  EMMS  coils  were  deployed  in 
the  narrow  (0.5  m  wide)  orientation,  and  0.25  m 
survey  lanes  were  used  to  maximize  coverage. 
As  shown  in  Figure  7,  coverage  in  open  areas 
was  relatively  good.  Coverage  in  wooded  areas 
was  unsatisfactory  with  relatively  large  areas 
left  unsurveyed.  Some  of  the  area  was 
inaccessible  because  of  system  design 
limitations.  The  difficulty  of  using  the  EMMS  in 
tight  areas,  combined  with  excessive  backpack 
loads,  difficulty  following  lane  layouts,  poor  site 
survey  management,  and  operator  frustration  all 
contributed  to  poor  performance. 

Survey  performance  in  the  woods  could  have 
been  improved  either  by  conducting  a  second 
survey  in  an  orthogonal  direction  to  fill  in  some 
of  the  missed  areas  or  by  using  the  bush  hog  to 
clear  remaining  brush  and  undergrowth  before  a 
resurvey.  In  reality,  however,  the  primary 
lesson  learned  from  the  EMMS  deployment  at 
the  E-Range  was  that  more  development  was 
required. 

4.3.2  Missed  Targets 


Figure  8.  Vehicular  (top  panel)  and  EMMS  (bottom 
panel)  EM  Surveys  from  the  L- Range. 


Primary  system  performance  was  measured  against  the  seeded  targets.  Secondary  considerations 
were  given  to  the  performance  of  the  man-portable  systems  relative  to  the  MTADS  vehicular 
system,  the  performance  of  the  rmn-portable  systems  using  GPS  versus  acoustic  navigation,  and 
the  performance  of  the  MMS  compared  with  the  EMMS.  The  MMS  detected  all  targets 
characterized  using  the  vehicular  magnetometer  array  and  with  one  exception,  detected  and 
analyzed  all  the  seed  targets  in  each  of  the  surveys  that  covered  their  positions.  (Seed  target  T- 
14  was  not  detected  by  the  MMS  because  it  fell  in  a  missed  survey  area.) 


In  the  EMMS  survey  using  GPS  navigation,  14  targets  were  missed  that  were  characterized  in 
the  vehicular  EM  survey  and  in  all  the  magnetometer  surveys.  Coverage  by  the  EMMS  in  the 
open  range  area  was  good.  In  every  case,  a  target  was  missed  because  either  there  was  no 
measurable  signal  in  the  EM  data  or  the  signal-to-noise  ratio  placed  the  target  below  the  imit 
required  for  analysis.  The  navigation  error  caused  by  the  rocking  of  the  GPS  antenna  created  an 
additional  noise  source,  which  tended  to  smear  out  a  target  signal  and  make  it  very  ragged.  (See 
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Figure  8.)  In  some  cases,  this  created  a  situation  in  which  the  fit  algorithm  would  not  converge. 
In  other  cases,  it  was  difficult  to  identify  a  target  signal  in  the  relatively  noisy  background. 

Performance  evaluation  on  the  entire  range  area  can  be  made  by  comparing  the  MMS  and 
EMMS  surveys  using  the  acoustic  navigation  system.  The  MMS  survey  contains  21  targets  that 
were  undetected  in  the  EMMS  survey.  Of  the  21  missed  targets,  9  targets  (in  the  woods)  were  in 
areas  missed  by  the  EMMS  survey.  In  the  remainder  of  the  cases,  the  EM  signal  was 
undetectable  or  too  weak  for  analysis. 

4.3.3  Target  Location 

Table  5  summarizes  the  performance  of  the  various  systems  in  pinpointing  the  positions  of  the 
seed  targets.  The  ground  truth  positions  are  those  measured  by  the  surveyor  at  the  time  the 
targets  were  buried. 

Table  5.  Errors  in  Predicted  Location  and  Depth  for  Seed  Targets  at  the  L-Range. 


Survey 

Vehicular 

Mag/GPS 

Vehicular 

EM/GPS 

MMS/ 

GPS 

MMS/Acoustic 

EMMS/ 

GPS 

EMMS/Acoustic 

Horizontal  error  (cm) 

11 

21 

11 

43 

26 

60 

Depth  error  (cm) 

5 

29 

5 

11 

(*) 

(*) 

(*)  At  the  time  of  the  demonstration,  the  fitting  algorithms  in  the  DAS  were  not  calibrated  with  the  new  coil  signals  to  provide  meaningful  depth 
predictions. 


The  location  and  depth  prediction  capabilities  of  the  vehicular  MTADS  systems  were  equivalent 
to  that  system’s  performance  in  many  prior  demonstrations.  Targets  are  typically  located  to 
within  10-15  cm  with  the  magnetometer  array  and  20-25  cm  with  the  EM  array.  Depth 
predictions  of  the  MTADS  DAS  based  on  magnetometer  data  are  typically  within  the  volume  of 
the  UXO  item.  Depth  predictions  using  the  point  dipole  algorithm  with  EM  data  are  less 
reliable.  The  vehicular  magnetometer  survey  and  the  MMS/GPS  survey  results  demonstrate  that 
the  MMS  can  provide  equivalent  field  performance  to  the  vehicular  system. 

4.3.4  Acoustic  Navigation  System 

Use  of  the  acoustic  navigation  system  degrades  the  location  accuracy  relative  to  the  GPS  system. 
With  careful  use,  however,  position  uncertainties  of  0.25  m  with  the  acoustic  navigation  system 
can  be  used  to  hold  target  location  predictions  to  <0.5  m.  In  conducting  acoustic  navigation 
surveys  in  open  areas,  site  dimensions  should  not  exceed  200  feet  in  any  direction.  Antenna 
locations  should  surround  the  perimeter  of  the  site,  and  it  is  helpful  to  have  an  antenna  centrally 
located  within  the  site.  System  performance  degrades  when  there  is  high  background  noise 
interference.  Even  wind  blowing  through  knee-high  grass  raises  the  operational  threshold  of  the 
antennas  (to  exclude  noise)  to  a  level  that  noticeably  degrades  system  performance. 

A  single  antenna  setup  was  used  to  survey  the  wooded  part  of  the  E-Range.  This  setup  worked 
satisfactorily  even  though  many  of  the  antennas  were  not  directly  (line- of- sight)  visible  to  each 
other  because  of  elevation  changes  or  because  trees  and  brush  obstructed  visibility.  A  careful 
initial  setup  is  required.  Three  of  the  antennas  should  be  at  known  coordinates,  and  it  is 


21 


recommended  that  system  performance  be  evaluated  against  a  known  position  within  the  survey 
area  before  undertaking  a  survey.  Even  though  this  system  was  difficult  to  use,  it  was  the  best 
available  (non- GPS)  automated  navigation/location  system  that  could  be  used  to  conduct 
digitally  mapped  UXO  surveys. 

4.3.5  Ergonomics 

Mechanically,  the  MMS  performed  satisfactorily  in  the  L-Range  demonstration,  and  the  system 
performance  as  a  UXO  survey  tool  met  all  system  design  specifications.  However,  the  system 
was  not  appropriate  for  transition  to  commercial  applications  where  it  would  be  used  in  more 
demanding  conditions  by  less  skilled  operators. 

EMMS  system  performance  was  disappointing  on  several  levels.  Cost  constraints  in  system 
development  and  integration  led  to  use  of  existing  equipment  that  was  too  heavy  and  bulky  for 
backpack  applications.  The  system  design,  based  on  a  narrow  coil  mounted  on  a  trailing  cart 
supported  by  high  wheels,  had  poor  usability.  The  electronic  performance  of  the  newly 
purchased  coil  systems  proved  to  be  too  insensitive  for  dependable  UXO  surveys  for  either 
shallow  or  deep  targets.  A  complete  redesign  of  the  EMMS  was  undertaken  in  preparation  for 
the  final  demonstration. 
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5.0  TECHNOLOGY  MODIFICATIONS  PRIOR  TO  JPG-V 
DEMONSTRATION 

5.1  COMPONENT  CHANGES  IN  THE  FINAL  PROTOTYPES 

Following  the  L-Range  demonstration,  additional  system  development  and  a  new  system 
demonstration  was  required  to  produce  MTADS  man-portable  adjuncts  that  could  be 
commercialized.  ESTCP  supported  this  development.  The  final  prototypes  were  demonstrated 
during  field  surveys  at  the  Jefferson  Proving  Ground  in  the  summer  of  2000  [6,  18]. 

5.2  SUBSYSTEM  COMPONENT  UPGRADES 

Each  system  subcomponent  was  reconsidered  from  a  requirements  and  performance  perspective 
[18].  Only  one  significant  change  was  made  to  the  system  performance  specifications.  Given 
the  difference  in  performance  of  the  vehicular  EM  arrays  and  what  would  become  the  final  man- 
portable  EM  array,  it  was  unrealistic  to  attempt  to  develop  the  DAS  so  that  man-portable  and 
vehicular  EM  survey  data  could  be  used  interchangeably.  The  DAS  was  modified  to  separately 
accept  and  fit  either  vehicular  or  EMMS  data,  but  no  attempt  was  made  to  create  a  conversion 
that  would  make  the  two  data  sets  equivalent. 

5.2.1  Cabling 

There  were  too  many  cables  and  connectors  used  to  interconnect  system  components  (batteries, 
sensors,  navigation  components  and  data  logger)  and  they  were  not  rugged  enough  for  sustained 
field  work,  so  flie  number  of  cables  was  reduced.  New  cables  with  improved  strain- relief 
connectors  were  designed  for  connection  to  interface  plates  in  the  new  backpacks. 

5.2.2  Batteries 

Each  component  of  the  man-portable  prototypes  was  delivered  by  its  manufacturer  with  an 
individual  battery  pack.  The  capacity  of  these  battery  packs  ranged  from  several  days  for  the 
acoustic  location  system  to  1.25  hours  for  the  EM61.  In  operation,  this  meant  that  EM  battery 
change-out  occurred  every  hour  while  carrying  other  heavy  long-hfe  batteries.  New  battery 
requirements  were  established  that  required  1-hour  performance  with  a  25%  excess  capacity. 
Powering  the  new  EM  coils  required  a  new  lead- acid  battery,  which  was  placed  on  the  system 
cart  as  discussed  below.  Battery  chargers  and  batteries  were  purchased  to  permit  day-long, 
continuous  surveys. 

5.2.3  Data  Logger 

The  original  modifications  to  the  Geometries  858  data  recorder  greatly  increased  its  flexibility 
and  usability  with  the  MMS  and  the  EMMS.  However,  the  operator  did  not  have  real-time 
display  access  to  GPS  fix  quality.  Eor  the  E-Range  demonstrations,  fix  quality  requirements 
were  accommodated  by  planning  to  survey  when  good  satellite  visibility  was  assured.  In  a  real- 
world  environment,  this  was  unacceptable.  Additional  modifications  to  the  data  logger  were 
made  to  display  the  GPS  fix  quality  and  to  sound  an  alarm  when  the  quality  dropped  below  Eix 
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Quality  3.  The  enhanced  Geometries  858  data  logger  is  now  commercially  available  from 
Geometries. 

5.2.4  MMS  Sensor  Cart 


The  design  of  the  magnetometer  cart  br  the  L- Range  demonstration  with  its  mounts  for  the 
antenna/transponder,  introduced  ease  of  use  and  minimal  weight.  Although  the  cart  performed 
acceptably,  the  tests  were  not  in  difficult  terrain.  Subsequent  analysis  showed  that  a  major 
increase  in  strength  and  reliability  could  be  obtained  with  a  small  weight  penalty. 


The  magnetometer  cart  was  redesigned  for 
improved  ruggedness  and  ergonomics.  The  new 
cart,  shown  in  Figure  9  with  the  hood  on  and  in 
Figure  10  with  the  hood  off,  is  constructed  of  a 
combination  of  fiberglass  and  plastic 
components.  With  wheels  side  by  side  between 
the  sensors,  this  design  works  better  in  tight 
spaces  and  on  rugged  and  uneven  terrain.  The 
hood  protects  the  sensors  and  the  antenna  and 
allows  the  system  to  slide  through  grass  and  low 
brush  without  snagging  the  cart  or  sensors.  The 
height  of  the  handles  and  the  weight  that  the 
operator  has  to  carry  in  his  hands  represents  a 
considerable  ergonomic  improvement  over  earlier 
designs.  The  complete  system  can  be  operated  by 
a  single  operator  for  an  extended  time  without 
tiring.  MMS  weighs  52  pounds  with  the  hood  and 
38  pounds  without.  The  total  backpack  weight  is 
18  pounds. 


Figure  9.  MMS  Deployed  with  Protective 
Hood  and  GPS  Navigation  Hardware. 


In  the  case  of  the  EM61,  deploying  the  prototype 
with  the  wheels  provided  by  the  manufacturer 
positions  the  transmit  coil  40  cm  above  the 
ground.  Because  the  measured  EM  signal  falls  off 
as  1/r^,  a  large  increase  in  signal  (and  thus  Figure  10.  MMS  Deployed  on  Area  3  at  JPG. 
sensitivity  for  relatively  shallow  objects)  can  be 

obtained  by  lowering  the  sensor.  Eollowing  delivery  and  acceptance  of  the  new  EM  sensor,  a 
new  EM  cart  was  designed  and  built  to  support  the  system.  It  is  more  rugged,  more  pitch  and 
roll  stable,  provides  better  weight  balance  in  front  of  and  behind  the  wheels  to  minimize  carry 
weight  for  the  operator,  and  incorporates  a  new  design  that  put  the  sensors  in  front  of  the 
operator. 


5.2.5  EMMS  Sensor  Cart 


The  design  of  the  sensor  trays  permits  the  sensor  to  be  mounted  with  the  long  dimension  either 
across  the  track  or  pointing  down  the  track,  as  shown  in  Eigure  11.  The  new  sensor  coils  require 
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a  heavy  lead- acid  battery  for  power.  The 
system  is  designed  to  collect  data  for  1  hour 
before  battery  change  out.  Depending  on  the 
orientation  of  the  coils,  the  battery  is  mounted 
to  minimize  the  signal  from  the  plates  at  the 
receive  coils  and  to  balance  the  weight  over  the 
wheels.  The  weight  the  operator  has  to  lift  is 
not  large,  but  the  weight  of  the  total  system, 
including  battery,  cart,  coils,  and  antenna,  is 
110  pounds,  and  pushing  this  load  proved 
difficult  in  rugged  terrain  or  in  deep  mud.  The 
redesigned  backpack  and  system  components 
permit  the  operator  to  carry  all  components  in  a 
31 -pound  backpack. 


Figure  11.  EMMS  Shown  with  Digitai 
Inciinometer  and  GPS  Navigation  Hardware. 


The  MMS  and  EMMS,  when  deployed  with  the 
acoustic  navigation  system,  have  the  ceramic 
transmitter  deployed  directly  above  the  sensors. 

The  transmitter  does  not  interfere  with  either 
sensor.  The  transmitter  is  shown  deployed  on 
the  EMMS  in  Eigure  12. 

5.2.6  EM  Coil 

Geonics  developed  a  new  0.5  m  X  1.0  m  coil 
system  that  is  more  sensitive  than  any  prior 
EM61  system.  Changes  in  coil  design,  power 
delivery  system,  and  detection  electronics 
improved  both  the  transmit  power  and  the 
signal  detection  sensitivities  of  the  sensor.  The  coil,  as  currently  deployed  on  the  EMMS,  can  be 
purchased  as  a  standard  product  from  the  manufacturer.  The  increased  transmit  power  required  a 
substantial  increase  in  battery  capacity.  The  Powersonic  Model  12330  (33  amp-hours)  met  the 
increased  requirement.  A  battery  charging  system  allows  five  batteries  to  be  charged 
simultaneously,  permitting  continuous  operation  of  the  EMMS  for  a  full  day. 


Figure  12.  EMMS  Deployed  with  the  Acoustic 
Navigation  Transponder. 


5.2.7  DAS 


An  upgrade  to  the  MTADS  data  analysis  system  accommodates  new  developments  in  this 
program  as  well  as  other  ESTCP  UXO  programs.  The  current  DAS  has  switches  to  allow  target 
analysis  of  survey  data  with  algorithms  tailored  for  the  new  coil.  As  part  of  the  shakedown 
process  for  the  new  MMS  and  EMMS  instruments,  new  data  sets  were  obtained  from  the  test  pit 
for  a  wide  range  of  inert  ordnance  items.  These  data  sets  were  incorporated  into  the  ordnance 
library. 
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6.0  JPG-V  DEMONSTRATION  DESIGN 


ESTCP  funded  technology  demonstrations  at  the  JPG  in  Madison,  Indiana,  at  three  1- hectare 
areas  near  the  40-acre  site  used  during  the  JPG  Phase  IV  demonstrations  [16].  The  technical 
objective  of  this  demonstration  [19]  was  to  evaluate  detection  and  discrimination  capabilities  and 
cost  and  production  rates  of  three  advanced  UXO  systems  in  magnetic  clutter  environments  such 
as  those  encountered  at  Kaho’olawe,  Hawaii. 

This  demonstration  was  intended  to  collect  information  from  this  limited  range  of  test  scenarios 
to  quantify  the  advantages  and  disadvantages  of  each  of  the  three  technologies.  A  long-term 
objective  of  this  demonstration  was  to  provide  high-quality,  georeferenced  data  to  support  sensor 
development  and  improvements  in  UXO  analysis  technologies. 

The  three  demonstrated  technologies  include: 

•  The  Geophex  Ltd.  GEM- 3,  a  multichannel  frequency  domain  EMI  sensor  system 
operated  by  Geophex  Ltd.  personnel  with  processing  support  from  AETC  Incorporated. 

•  The  NRL  Man-Portable  EMMS  adjunct[20]  to  the  MTADS,  a  single  time-channel  time 
domain  EMI  sensor  operated  by  personnel  from  NRL  with  processing  support  from 
AETC  Incorporated. 

•  The  Geonics  Ltd.  EM63,  a  multichannel  time  domain  EMI  sensor  operated  by  personnel 
from  NAEVA  Geophysics,  Inc. 

Each  of  the  three  sensors  was  integrated  into  a  man-portable  platform  that  included  data 
acquisition/storage  and  differential  GPS  receivers. 

6.1  PERFORMANCE  OBJECTIVES 

The  performance  objectives  of  the  demonstrations  were  as  follows  [19]: 

•  Evaluate  demonstrators’  detection  and  discrimination  capabilities  under  realistic  target, 
geologic  clutter,  man-made  clutter,  and  topography  scenarios  while  operating  as 
efficiently  and  cost-effectively  as  possible. 

•  Evaluate  demonstrators’  ability  to  (a)  analyze  survey  data  on-site;  (b)  provide  prioritized 
“dig  lists”  that  include  detection,  discrimination,  and  identification  estimates  with 
associated  confiderce  levels;  and,  (c)  provide  georeferenced  anomaly  maps. 

•  Collect  manpower,  time,  and  cost  data  for  all  tasks. 

•  Compare  the  performance  of  the  systems  with  baseline  mag- and-flag  technology. 
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•  Provide  high  quality,  ground -truthed,  georeferenced  data  for  post- demonstration  analysis 
and  for  use  by  other  government,  university,  and  industry  researchers  to  develop 
improved  models  and  analysis  technologies. 

Operations  and  maintenance  costs  include  labor  costs  associated  with  setup,  calibration,  survey, 
analysis,  and  maintenance,  as  well  as  any  required  support  equipment,  consumables,  and 
supplies.  For  evaluation  purposes,  all  demonstrators  used  the  following  (loaded)  labor  rates: 

•  Supervisor — $95/hour 

•  Data  Analyst — $57/hour 

•  Logistics/field  support — $28. 50/hour 

In  addition  to  the  costs  listed  above,  cost  penalties  were  imposed  for  false  positives  and  for  UXO 
misclassified  as  clutter. 

6.2  TEST  SITE  CHARACTERISTICS 

Figure  13  shows  magnetic  anomalies 
from  data  collected  by  the  MTADS 
vehicular  system  in  Area  1,  which  was 
selected  because  it  contains  high 
magnitude  magnetic  anomalies  from 
geologic  sources  that  cover  a  fairly 
large,  contiguous  area.  The  long 
magnetic  anomaly  (red  area)  appearing 
near  the  center  of  Area  1  represents 
variations  from  the  background  mean  of 
-1-150  nT  to  -100  nT.  Area  1  has  sparse 
tree/shrub  coverage  and  its  topography 
includes  rolling  terrain  and  ditches. 

Area  1  was  seeded  with  the  largest 
concentration  of  UXO  and  clutter  items; 
a  substantial  number  were  placed  within 
the  high  magnetic  background  locations. 


Figure  14  shows  magnetic  anomalies 
from  data  collected  by  the  MTADS 
vehicular  system  in  Area  2,  which  was 
chosen  because  it  has  a  significant 
number  of  magnetic  geologic  anomalies  (the  patchy  red  areas).  The  geological  interferences  in 
this  area  are  more  compact  and  lower  in  magnitude  (-1-  35  nT),  thus  providing  a  different 
interference  problem  from  that  of  Area  1 .  The  topography  in  Area  2  also  includes  rolling  terrain 
and  a  small  ravine.  Area  2  was  seeded  with  a  smaller  number  of  UXO  and  clutter  items  than 
Area  1. 
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Figure  13.  Magnetic  Anomalies  in  Area  1  from  Data 
Collected  by  the  MTADS  Vehicular  System. 
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Figure  15  shows  magnetic  anomalies  from 
data  collected  by  the  MTADS  vehicular 
system  in  Area  3,  which  was  chosen 
because  it  has  very  low  amplitude 
geologic  interferences  and  very  flat 
terrain.  This  area  has  a  variation  from  the 
mean  background  of  only  +  6  nT.  Area  3 
was  seeded  with  the  fewest  UXO  and 
clutter  items. 

6.3  PHYSICAL  SETUP  AND 
OPERATION 

6.3.1  Physical  Setup 

The  inert  UXO  items  emplaced  for  these 
demonstrations  include  all  the  items 
present  at  the  Kaho’olawe  quality  control 
(QC)  range  with  the  exception  of  large  air- 
delivered  bombs.  The  UXO  items  ranged 
from  20- mm  projectiles  buried  near  the 
surface  to  155-mm  projectiles  buried  up  to 

1.2  m  deep.  Clutter  items  emplaced  ranged 
from  small  (less  than  0.5  kg)  to  large  (up  to 
5  kg)  munitions  fragments  and  other  man¬ 
made  clutter  such  as  horseshoes  and  metal 
banding.  In  addition,  magnetic  rocks 
(basalt)  and  soils  obtained  from  sites  in 
Kaho’olawe  and  other  U.S.  sites  were 
emplaced. 

Burial  depths  of  the  inert  munitions  used  for 
field  tests  at  JPG  did  not  exceed  those 
specified  for  the  Kaho’olawe  Tier  II 
clearance  requirements.  For  the  tests  at 
JPG,  the  20-mm  targets  were  emplaced 
without  the  casing,  and  burial  depths  ranged 
from  flush  with  the  surface  to  15.2  cm. 

6.3.2  Operation 

Area  3  was  staked,  gridded  and  surveyed 
with  the  EMMS  on  August  28,  2000.  Area 
1  was  staked,  gridded,  and  surveyed  with 
the  EMMS  on  August  29,  2000.  Area  2  was  similarly  completed  on  August  30,  2000.  The  setup 
and  survey  production  rates  are  summarized  in  Table  6. 
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Figure  14.  Magnetic  Anomalies  in  Area  2  from  Data 
Collected  by  the  MTADS  Vehicular  System. 
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Figure  15.  Magnetic  Anomalies  in  Area  3  from  Data 
Collected  by  the  MTADS  Vehicular  System. 
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Table  6.  Production  Rates  Associated  with  Various  JPG-V  Survey  and  Analysis  Activities. 


Survey 

Area  3  (Man-Hours) 

Area  2  (Man-Hours) 

Area  1  (Man-Hours) 

Total  (Man-Hours) 

Stake  and  grid 

9.5 

9.2 

10.3 

29 

EMMS  survey 

18 

21.5 

19.3 

59 

EMMS  analysis  (on-site) 

20.5 

18.8 

23 

62.5 

Target  reports 

23.5 

MMS  survey 

20 

20 

Vehicular  mag  survey 

4.75 

3.5 

4.5 

12.8 

Clean  up /pack  out 

15 

6.4  SAMPLING  AND  MONITORING  PROCEDURES 

The  NRL  predictions  of  ordnance  type  were  grouped  into  categories  of  similarly  sized  items 
(e.g.,  57-/60- mm,  76-/81- mm,  105- mm/4.2- in,  152-/155- mm).  In  some  situations,  predicted 
sizes  fell  between  more  disparate  items  such  as  60-/8 1-mm  mortars.  Type  predictions  attempted 
to  reflect  these  uncertainties.  In  general,  size  predictions  resulting  from  combined  EM  and 
magnetometer  analyses  are  more  precise  and  reliable  than  those  resulting  from  EM  data  alone. 

6.5  ANALYTICAL  PROCEDURES 

The  evaluation  factors  and  procedures  related  to  this  demonstration  include: 

1.  Equipment  setup,  calibration  time,  and  man-hour  requirements 

2.  Actual  survey  time  and  man-hour  requirements  for  each  of  the  three  test  areas 

3.  Downtime  because  of  system  malfunctions  and  maintenance  requirements 

4.  Reacquisition/resurvey  time  and  man-hour  requirements 

5.  Actual  data  processing/analysis  time  and  man-hour  requirements  (all  performed  on-site) 

6.  Prioritized  dig  lists  with  associated  confidence  levels 

7.  Discrimination  capability — ability  to  separate  detected  anomalies  into  UXO  and  non- 
UXO  objects 

8.  Identification  capability — ability  to  classify  UXO  targets  by  class  (e.g.,  mortar, 
projectile)  and  type  (e.g.,  152- mm 

9.  Predicted  target  location  accuracy,  including  depth  estimates 

10.  Georeferenced  anomaly  maps 

1 1 .  Probabilities  of  detection  (Pd) 

12.  Ealse  alarm  rates  (PAR) 
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The  method  for  determining  and  documenting  the  first  three  items  involved  on-site  government 
representatives  tracking  and  recording  the  number  of  personnel  and  time  spent  performing  each 
task.  If,  during  data  analysis,  the  demonstrator  determined  a  need  to  resurvey  any  part  of  the  test 
areas  or  any  previously  detected  anomalies,  all  setup,  calibration,  survey,  downtime, 
reacquisition  times,  and  man-hour  requirements  were  recorded  individually  (as  in  items  1 
through  3),  but  were  compiled  separately  as  reacquisition/resurvey  time  (item  4). 

To  evaluate  item  5,  the  government  required  that  all  data  processing  and  analysis  tasks  required 
to  produce  items  6  through  10  be  conducted  in  the  JPG  office  trailer,  and  that  no  data  be  taken 
off-site  until  these  items  were  submitted  to  the  on-site  government  representative. 
Demonstrators  provided  all  computer  hardware,  software,  and  support  equipment  needed  to 
produce  the  required  analysis  products. 

Development  and  evaluation  of  items  6  through  10  were  as  follows. 

•  Each  demonstrator  was  required  to  provide  two-dimensional  (2-D)  anomaly  maps  of  each 
1 -hectare  area.  The  anomalies  were  tabulated  into  a  dig  sheet  for  each  test  area, 
organized  to  ensure  as  high  a  Pd  as  possible  for  the  full  range  of  UXO  targets. 

•  Each  anomaly  in  each  list  contained  the  (x,  y)  location  and  an  estimated  burial  depth;  an 
attempt  to  separate  (discriminate)  UXO  from  clutter  items  and  to  identify  UXO  by  class 
and  type;  and  a  ranked  list  in  the  following  order: 

1 .  UXO — high  confidence 

2.  UXO — medium  confidence 

3.  UXO — low  confidence 

4.  Clutter — low  confidence 

5.  Clutter — medium  confidence 

6.  Clutter — high  confidence 

•  Each  demonstrator  was  then  required  to  specify  a  threshold  on  each  prioritized  list  where 
he  would  recommend  that  all  objects  at  or  above  that  threshold  be  excavated  and  those 
below  be  left  in  place.  To  add  realism  to  this  discrimination  decision,  the  following  cost 
factors  were  applied: 

❖  Eor  every  clutter  item  selected  for  “digging,”  a  $200  cost  penalty  was 
assigned. 

❖  To  reflect  the  unacceptable  risk  of  leaving  UXO  in  the  ground,  if  one  or 
more  UXO  items  were  placed  in  the  “no  dig”  portion  of  the  list,  it  was 
assumed  that  the  grid  (i.e.,  the  entire  1-hectare  area)  failed  QA  and  a  cost 
penalty  equal  to  the  cost  of  a  resurvey  was  assigned. 

❖  One  or  more  missed  targets  in  each  area  were  also  assigned  a  cost  factor 
equal  to  the  cost  of  a  resurvey. 
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Pd  and  FAR  values  were  calculated  from  the  prioritized  dig  lists  as  follows:  Maximum 
achievable  Pd  for  each  area  was  calculated  as  the  number  of  items  in  the  entire  list  that 
correspond  to  emplaced  UXO  targets  (even  though  they  may  have  been  misclassified  as  clutter) 
divided  by  the  actual  number  of  UXO  targets  emplaced  in  that  site.  A  receiver  operating 
characteristic  (ROC)  curve  was  developed  by  varying  the  dig  threshold  and  computing  Pd  and 
FAR  at  each  increment  until  the  maximum  Pd  was  reached. 

After  the  demonstrators  submitted  the  dig  sheets  described  above,  they  were  given  the 
opportunity  to  reanalyze  the  data  to  develop  prioritized  dig  sheets  that  took  into  account  only 
targets  larger  than  20- mm  projectiles  (20- mm  projectiles  were  assumed  to  be  clutter  for  the 
evaluation). 
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7.0  JPG-V  PERFORMANCE  ASSESSMENT 


7.1  JPG-V  DATA 

Georeferenced  anomaly  maps 
produced  by  the  EMMS  in  Areas  1, 

2,  and  3  are  shown  in  Figure  16, 
Figure  17,  and  Figure  18, 
respeetively.  The  only  overall 
conclusion  that  can  be  derived 
from  these  maps  is  that  the  EMMS 
demonstrated  the  eapability  to 
suppress  the  high  magnetic  _ 
background  from  geologic  sources,  •§• 
and  the  capability  to  provide  high- 
quality  (well-  localized,  high 
signal-to-background  target 

signatures)  georeferenced  data. 
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Analyses  of  field  survey  data  were 
performed  in  three  stages.  The 
data  were  initially  analyzed  before 
leaving  the  JPG  test  site  to  produee 
three  prioritized  dig  lists  (one  for 
eaeh  survey  area)  eontaining  all 
anomalies  investigated.  The  data 
were  also  analyzed  off- site  to 
produce  additional  prioritized  dig  lists  that  ineluded  only  targets  estimated  to  be  larger  than  a  20- 
mm  projeetile  ^.e.,  the  smallest  objeet  of  interest  eorresponded  to  a  57-mm  projeetile).  After 
submitting  the  second  set  of  dig  lists,  the  demonstrators  were  provided  with  the  MTADS  mag 
data  and  requested  to  submit  two  additional  sets  of  prioritized  dig  lists  (with  and  without  20- mm) 
that  ineluded  eombined  mag  and  EM  analysis  results. 
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Figure  16.  EM  MS  Survey  of  Area  1  atj  PG. 


A  number  of  post- demonstration  adjustments  to  the  ground  truth  were  neeessary  in  order  to 
accurately  account  for  anomalies  resulting  from  metallic  objects  that  were  neither  detected  nor 
emplaced  during  the  site  preparation  for  this  demonstration.  After  initial  evaluation  of  the 
submitted  dig  lists,  it  became  apparent  that  each  demonstrator  declared  targets  at  locations  where 
no  items  had  been  emplaced  and  where  the  magnetic  anomalies  from  geologic  sources  were  not 
significant.  A  decision  was  made  to  excavate  those  locations  within  the  three  test  areas  where 
two  or  more  demonstrators  had  deelared  UXO  targets. 
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Digging  revealed  that  in  Area  3  (the 
site  north  of  the  former  40- acre  site 
during  prior  JPG  demonstrations)  all 
the  declarations  corresponded  to  farm- 
related  ferrous  objects  such  as 
portions  of  horseshoes,  plow  points, 
and  harness  hardware  and  did  not 
include  any  items  from  previous  JPG 
demonstrations.  As  a  result,  it  was 
decided  to  include  all  declarations  not 
corresponding  to  emplaced  items  in 
Area  3  as  false  alarms  because  of  non- 
UXO  ferrous  objects. 

On  the  other  hand,  in  Areas  1  and  2 
(which  were  inside  the  40- acre  site), 
limited  digging  revealed  a  number  of 
inert  UXO  remaining  from  previous 
JPG  demonstrations,  including 
projectiles,  mortars,  flares,  and 
fabricated  clutter  items.  Even  though 
JPG- IV  ground  truth  had  been  used  to 
clear  these  areas,  items  emplaced 
during  earlier  demonstrations  had 
remained.  As  a  result,  the 
government  examined  JPG- 1  through 
JPG- III  ground  truth  and  identified 
items  that  matched  the  locations  of 
anomalies  declared  by  any  one  of  the 
demonstrators  as  UXO  targets.  These 
objects  were  then  removed  from  the 
evaluation  of  results.  All  other  UXO 
target  declarations  that  did  not 
correspond  to  items  that  were 
emplaced  as  part  of  this  ESTCP 
project,  and  that  were  not  included  in 
the  ground  truth  from  prior  JPG 
demonstrations,  were  evaluated  as 
false  alarms.  The  option  of  limiting 
the  evaluation  to  only  the  objects 
emplaced  for  this  demonstration  was 
considered  and  rejected  because  it 
would  defeat  the  primary  objective  of 
the  test,  which  was  to  evaluate  system 
performance  in  high  natural  magnetic 
background  environments. 
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Figure  17.  EMMS  Survey  of  Area  2  at  JPG. 
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Figure  18.  EM  M  S  Survey  of  Area  3  at  J  PG. 
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One  of  the  critical  evaluation  factors  for 
this  demonstration  is  the  detection 
performance  of  advanced  UXO  detection 
systems.  The  metrics  used  to  quantify 
the  detection  performance  consist  of  the 
pseudo- ROC  curves,  the  single-point 
Pd/FAR,  and  the  maximum  achievable 
Pd-  The  methods  used  to  estimate  these 
metrics  from  the  prioritized  dig  lists 
were  described  in  Section  6.  Briefly,  the 
pseudo- ROC  curve,  which  graphically 
represents  the  tar^t  detection 
percentage  versus  the  number  of  false 
alarms  (or  false  alarm  rate  in  number  of 
false  alarms  per  hectare),  is  calculated 
by  sequentially  moving  from  the  top  of 
the  prioritized  dig  list  (i.e.,  the  highest 
confidence  UXO  target  declaration)  and 
determining  if  each  object  on  the  list 
(whether  classified  as  a  UXO  target  or 
clutter)  corresponds  to  an  emplaced 
UXO  target  location  (a  detection)  or  not 
(a  false  alarm).  The  single-point  Pd/FAR 
performance  is  based  on  the  point  on  the 
pseudo- ROC  curve  that  corresponds  to 
the  contractor- specified  dig  point  on  the 
prioritized  dig  list,  and  the  maximum 
achievable  Pd  is  based  on  the  highest 
point  on  the  pseudo- ROC  curve. 
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These  performance  metrics  are  presented 
in  Figure  19,  Figure  20,  Figure  21,  and 
Figure  22.  The  single- point  Pd/FAR  rate 
is  shown  as  a  yellow  triangle  on  the 
pseudo- ROC  curve.  The  red  diamond 

corresponds  to  the  single- point  Pd/FAR  performance  point  of  the  mag- and- flag  survey. 


Figure  19.  Pseudo-ROC  Curves  for  EMMS  Surveys  of 
Areas  1  Through  3  and  Analysis  That  Includes  20-nun 
Projectiles  hut  Excludes  MTADS  Mag  Data. 
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Several  points  must  be  kept  in  mind  when 
interpreting  these  plots.  The  abscissa  in 
the  pseudo-ROC  curves  is  not  the 
probability  of  a  false  alarm  (Pfa),  but  rather 
total  number  of  false  alarms.  As  a  result, 
the  absolute  slope  of  the  curve  has  no 
intrinsic  meaning  but  is  nevertheless 
useful  for  comparing  relative  performance 
among  different  systems.  These  curves 
combine  detection  and  discrimination  of 
ordnance  from  nonordnance.  Thus,  the 
initial  pseudo-ROC  curve’s  slope 
represents  the  anomalies  that  the 
demonstrator  has  declared  as  UXO  with 
the  highest  confidence.  A  flat  slope  in  this 
area  indicates  poor  discrimination 
capability.  The  final  slope  of  the  pseudo- 
ROC  curve  represents  anomalies  that  the 
demonstrator  declared  as  clutter  with  high 
confidence.  A  positive  slope  indicates 
there  are  UXO  targets  that  the 
demonstrator  would  leave  unexcavated. 


In  Figure  19,  the  steep  early  slopes  of  the 
pseudo-ROC  curves  indicate  significant 
discrimination  capability.  The  EMMS 
outperformed  the  mag- and- flag  system  at 
all  three  test  areas,  and  the  single-point 
performance  points  met  the  Kaho’olawe 
Tier  II  requirements.  Based  on  the 
maximum  of  the  pseudo-ROC  curves,  the 
EMMS  did  not  achieve  100%  detection  at 
any  of  the  three  sites. 
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The  naturally  occurring  geologic  magnetic 
noise  and  the  emplaced  magnetic  rocks 
presented  no  problems  to  the  EMMS.  All 
false  alarms  included  in  the  submitted  dig 
lists  were  attributable  to  metallic  clutter,  and 
analyses  of  the  georeferenced  maps  showed 
magnetic  rocks. 


Area  3 

Figure  20.  Pseudo-ROC  Curves  for  EMMS 
Surveys  of  Areas  1  Through  3  and  Analysis  That 
Excludes  Both  20  mm  Projectiles  and  MTADS  Mag 
Data. 

discernible  anomaly  over  any  of  the  emplaced 


Eigure  20  shows  the  detection  performance  of  the  EMMS  based  on  the  results  of  the  off- site 
analyses  that  excluded  objects  estimated  to  be  the  size  of  20- mm  projectiles  or  smaller.  The 
objective  of  this  analysis  was  to  determine  system  performance  based  on  the  more  commonly 
encountered  mid-sized  (57- mm  and  larger)  UXO  targets.  No  comparisons  with  mag- and- flag 
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results  are  included  in  these  figures 
because  analog  magnetometers  lack  the 
capability  to  record  the  sensor  data  for 
reanalysis. 

In  Figure  20,  comparison  of  these  results 
with  those  provided  on-site  indicates 
significant  improvement  in  pseudo- ROC 
curve-based  performance.  In  addition,  the 
EMMS  operating  Pd/FAR  points  improved 
substantially,  particularly  in  Area  1  where 
100%  Pd  was  obtained  with  only  55  false 
alarms.  Maximum  Pd  also  increased 
slightly  in  the  other  two  areas  and 
exceeded  the  Tier  II  requirements. 
Excluding  the  small  targets  resulted  in 
significant  performance  improvements  for 
the  EMMS. 

Figure  21  shows  the  performance  of  the 
EMMS  when  the  MTADS  mag  data  were 
added  to  the  analysis  and  all  targets  were 
considered.  Overall,  EMMS  Pd  detection 
performance  actually  decreased  with  the 
addition  of  the  mag  data.  Comparison  of 
these  results  with  those  in  Figure  19 
shows  that,  for  all  three  areas,  the 
pseudo-ROC  curve  performance  was 
lowered  when  the  mag  data  were 
included  in  the  analysis.  The  maximum 
Pd  was  also  slightly  lower  in  all  three 
areas. 

Figure  22  shows  the  detection 
performance  of  the  EMMS  systems 
against  57- mm  and  larger  targets  after 
integrating  magnetometer  data  into  the 
analysis.  The  purpose  of  this  analysis 
was  to  quantify  performance 
improvements  in  midsized  UXO 
detection  from  magnetometer  data  under 
varying  clutter  conditions. 
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Figure  21.  Pseudo-ROC  Curves  for  EMMS  Surveys  of 
Areas  1-3  and  Analysis  That  Includes  Both  20-mm 
Projectiles  and  MTADS  Mag  Data. 


Overall  performance  improved  very  slightly  from  the  EM-only  analysis  presented  in  Figure  20. 
The  pseudo-ROC  curve  performance  for  all  three  areas  was  only  slightly  better  than  the  EMI- 
only  performance.  The  operating  Pd  performance  was  worse  for  the  mag-EMI  case  because,  for 
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Area  1 


all  three  areas,  the  operating  PdS  were 
slightly  lower,  and  they  occurred  with 
higher  false  alarm  counts.  Though  lower, 
operating  PdS  still  met  or  exceeded  Tier  II 
requirements  in  all  three  areas.  The 
maximum  achievable  Pd  was  slightly 
lower  for  Areas  1  and  3  and  unchanged  for 
Area  2. 

In  general,  the  addition  of  mag  data  had 
very  little  effect  on  the  early  part  of  the 
pseudo-ROC  curve  (i.e.,  discrimination 
ability).  The  effect  on  the  operating  PdS 
was  minor.  The  primary  impact  on 
EMMS  results  was  an  increase  in  the 
number  of  anomalies  included  in  the  dig 
lists,  most  of  which  are  correctly  classified 
as  clutter. 

The  discrimination  and  identification 
capabilities  of  UXO  systems  greatly  affect 
the  cost  and  residual  risks  associated  with 
any  UXO  cleanup  operation. 

Discrimination  and  identification 
performance  is  based  on  UXO  versus 
clutter  and  UXO  type  declarations 
included  in  each  of  the  required  prioritized 
dig  lists.  The  results  presented  in  this 
section  have  been  adjusted  to  account  for 
UXO -related  items  that  remained  in  Areas 
1  and  2  from  previous  JPG 
demonstrations. 

To  facilitate  the  evaluation  of  detection, 
discrimination,  and  identification 
performance  of  the  EMMS,  as  well  as 
comparisons  with  mag- and- flag  results 
where  appropriate,  dig  list  information  is 
summarized  in  Table  7.  It  should  be  noted 
that,  in  this  table,  dig  list  declarations  were 
interpreted  to  accept  the  correct  choice  of 
the  first  two  choices  listed.  Eor  example, 

if  the  dig  list  specified  57- mm  projectile/60- mm  mortar  for  an  actual  mortar  target,  credit  was 
given  in  the  mortar  class. 


Area  2 


Figure  22.  Pseudo-ROC  Curves  for  EMMS  Surveys  of 
Areas  1  Through  3  and  Analysis  that  Excludes  20-mm 
Projectiles  and  Includes  MTADS  Mag  Data. 
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Table  7.  Ability  of  EMMS  to  Detect  and  Discriminate  Targets  by  Class 


System  Class 

Area  1 

Area  2 

Area  3 

Total  Score  | 

Actual 

Found 

Match 

Actual 

Found 

Match 

Actual 

Found  Match 

Actual 

Found 

Vlatch  1 

1  On-site  analysis  (includes  20-mm  and  excludes  MX  ADS  mag  data)  | 

EMMS  P 

■a 

16 

13 

10 

53 

48 

25 

M 

12 

12 

06 

09 

08 

Ba 

33 

32 

19 

R 

04 

04 

02 

02 

02 

01 

01 

01 

01 

07 

07 

04 

Explosive 

27 

19 

- 

16 

08 

- 

10 

04 

- 

53 

31 

- 

Ordnance 

12 

09 

- 

12 

11 

- 

09 

09 

- 

33 

29 

- 

Detection 

04 

03 

- 

02 

02 

- 

01 

0 

- 

07 

05 

- 

Technologies, 

Inc.  (EODT) 

M 

R 

1  Off -site  analysis  (excludes  20-mm  and  MTADS  mag  data)  | 

EMMS/Mag 

P 

17 

17 

10 

HD 

09 

04 

^H 

06 

05 

33 

32 

M 

12 

12 

09 

HD 

12 

06 

08 

05 

33 

32 

WA 

R 

04 

04 

02 

02 

01 

01 

01 

01 

07 

07 

04  1 

1  Off-site  analysis  (includes  20-mm  and  MTADS  mag  data)  | 

EMMS/Mag 

P 

27 

25 

15 

16 

13 

06 

10 

09 

03 

53 

47 

24 

M 

12 

11 

07 

12 

12 

05 

09 

08 

06 

33 

31 

18 

R 

04 

04 

0 

02 

02 

00 

01 

01 

01 

07 

07 

01 

1  Off-site  analysis  (excludes  20-mm  and  includes  MTADS  mag  data)  | 

EMMS  P 

17 

17 

12 

■El 

■a 

05 

03 

■a 

M 

12 

11 

07 

^■9 

■9 

08 

06 

■9 

R 

04 

04 

0 

■9 

01 

01 

01 

01  1 

Note:  P  =  Projectile,  M  =  Mortar,  R  =  Rocket 


Each  demonstration  area  ineluded  three  UXO  targets  that  had  elutter  items  in  proximity  so  that 
their  magnetie  and  EMI  signatures  would  overlap.  The  purpose  of  these  elosely  spaeed  targets 
was  to  evaluate  the  spatial  resolution  of  the  EMI  sensor  and  to  determine  the  robustness  of  the 
diserimination,  classifieation,  and  identifieation  teehniques  employed.  Table  8  summarizes  the 
ability  of  the  EMMS  to  diseriminate,  elassify,  and  identify  UXO  targets  in  proximity  to  elutter. 

Table  8.  EMMS  Discrimination  Capability  Against  Overlapping  Targets. 


Area 

Ground  truth 

Found 

Declared 

1 

2  -  Clutter  1 13  -  105-mm  projectile 

113 

P 

18-  Clutter  117  -  152 -mm  projectile 

18 

P 

184-  Clutter  121  -  155-mm  projectile 

121 

P 

2 

16-  Clutter  131  -  81 -mm  mortar 

131 

M 

88  -  Clutter  120  -  60- mm  mortar 

120 

M 

76-  Clutter  166  -  2.75-in  rocket 

166 

C-M 

3 

6  -  Clutter  68  -  60-mm  mortar 

6 

C-H 

50  -  Clutter  76  -  60-mm  mortar 

50 

C-M 

62  -  Clutter  80  -  8 1-mm  mortar 

80 

M 

Note:  P  =  Projectile,  M  =  Mortar,  R  =  Rocket,  C-L  =  Clutter-Low,  C-M  =  Clutter -Medium,  C-H  =  Clutter- High 


In  Area  1,  the  EMMS  correctly  located  two  of  the  three  UXO  targets  and  eorrectly  identified  all 
three  as  projeetiles.  In  Area  2,  the  EMMS  eorreetly  loeated  all  three  UXO  targets.  The  two 


39 


mortar  rounds  were  correctly  classified  as  such,  whereas  the  2.75-in  rocket  was  classified  as 
clutter  with  medium  confidence.  In  Area  3,  the  EMMS  correctly  located,  discriminated,  and 
identified  only  one  of  three  overlapping  targets. 

The  (x,y)  location  performance  of  the  EMMS  was  evaluated  by  comparing  each  item  in  the 
onsite  dig  list  with  the  ground  truth,  determining  the  closest  item  (within  1  m)  to  an  emplaced 
UXO  target  location,  and  computing  the  error.  Overall,  the  EMMS  system  demonstrated  an 
average  position  error  of  22.1  cm.  Most  of  the  targets  detected  by  the  EMMS  were  well  within 
the  0.5-m  error  radius  required  by  Kaho’olawe  cleanup  criteria. 

The  ability  of  the  EMMS  to  estimate  the  depth  of  the  UXO  targets  is  summarized  in  Table  9. 
These  results  indicate  that,  although  the  performance  varied  significantly  among  test  areas,  the 
mean  depth  estimation  errors  were  within  the  0.5-m  allowable  error. 

Table  9.  UXO  Target  Depth  Estimation  Performance. 


Area 

Minimum  Error  (m) 

Maximum  Error  (m) 

Mean  Error  (m) 

Standard  Deviation 

1 

0.01 

0.65 

0.23 

0.17 

2 

0.00 

0.86 

0.27 

0.24 

3 

0.01 

0.37 

0.16 

0.10 

7.2  JPG-V  ASSESSMENT 

Surveys  of  Areas  1  and  2  included  regions  with  significant  geological  activity.  The  motion- 
induced  noise  in  the  EM  image  of  Area  3  (Eigure  18)  resulted  from  surveying  across  shallow 
furrows  dating  from  when  the  area  was  in  cultivation  before  its  use  as  a  target  range.  The 
magnetic  soil  deposits  distributed  across  much  of  Area  2,  and  in  the  ravines  in  Area  3  did  not 
significantly  interfere  with  the  EM  survey  data.  Their  effect  on  the  magnetometry  data  was  more 
pronounced.  Target  analysis  of  magnetometry  data  in  these  areas  is  much  more  tedious, 
requiring  display  of  small  areas  with  constant  resetting  of  offsets  and  detrending  during  the 
fitting  of  individual  targets.  In  general,  with  very  careful  analysis,  10  nT  anomalies  could  be  fit 
in  regions  with  geological  magnetic  sail  offsets  of  200  nT.  Small  targets,  such  as  20-mm 
projectiles  or  60-mm  mortars,  at  depth,  had  a  higher  probability  of  going  undetected  in  these 
areas. 

7.2.1  Detection  Sensitivity 

Detection  probability  statistics  for  Area  3  are  summarized  for  all  three  sites  in  Table  10,  which 
shows  a  comparison  among  the  three  demonstrators  using  data  taken  from  the  presentation  of 
Dr.  Ernesto  Cespedes  at  an  ESTCP  workshop.  NRE  data  were  turned  in  on-site,  and  assumed 
20-mm  projectiles  were  not  UXO.  The  absolute  detection  sensitivities  of  the  EMMS  and  MMS 
systems  were  similar.  The  EM  sensors  had  a  greater  sensitivity  for  small  shallow  objects 
whereas  the  magnetometer  array  was  more  sensitive  for  large,  more  deeply  buried  objects.  The 
EM  sensors  also  detected  small  shallow  objects  more  efficiently  because,  using  the  overlapping 
1-m  coils,  surface  coverage  approached  100%.  It  was  possible  for  small  objects,  such  as  20-mm 
projectiles,  to  pass  between  the  magnetometers  undetected,  even  with  a  25-cm  separation.  To 
some  extent,  the  detection  sensitivity  was  also  a  function  of  target  picking.  Saturation  picking 
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detected  more  metal  objects,  but  without  an  efficient  discrimination  tool  for  analysis,  many  more 
objects  would  have  to  be  dug  using  this  approach.  All  three  demonstrators  appeared  to  have 
sufficient  detection  sensitivity  to  detect  the  full  range  of  UXO  items  buried  at  the  depths  of  this 
demonstration.  The  differences  in  detection  efficiency  of  Surveyors  B  and  C,  relative  to  the 
MTADS,  was  largely  a  matter  of  the  number  of  targets  chosen  for  declaration. 

Table  10.  Surveyor  Performance  at  the  JPG- V  Demonstration.  * 


Probability  of  UXO  Detection 


NRL 

Surveyor  B 

Surveyor  C 

Area  1 

95.3% 

100.0% 

90.7% 

Area  2 

96.7% 

100.0% 

93.3% 

Area  3 

100.0% 

100.0% 

90.0% 

Overall 

96.8% 

100.0% 

91.3% 

Objects  Declared 


Total  objects  in  dig  sheet 

NRL 

Surveyor  B 

Surveyor  C 

Area  1 

149 

177 

144 

Area  2 

147 

236 

206 

Area  3 

142 

202 

105 

*  This  table  was  adapted  from  Dr.  Ernesto  Cespedes  from  an  ESTCP  workshop  presentation. 


7.2.2  Missed  Targets 

Assuming  the  sensors  have  sufficient  sensitivity  to  detect  these  UXO  challenges,  the  number  of 
missed  targets  reflects  three  factors:  the  signal- to-noise  and  spatial  resolution  in  the  data,  the 
number  of  targets  chosen  for  declaration,  and  experience  in  extracting  target  signatures  from 
geological  and  scrap/clutter  interferences  in  the  data.  While  the  EM  sensors  are  much  less 
subject  to  geological  interferences  from  magnetic  soils,  the  spatial  resolution  of  the 
magnetometer  sensors  is  significantly  higher,  allowing  better  unclustering  of  adjacent  or  nearby 
targets.  This  unclustering  (or  cleanup  of  complex  target  signatures)  is  a  time-consuming  data 
processing  step  that  does  not  readily  lend  itself  to  software  automation.  The  time  expended  in 
this  data  cleanup  is  in  part  responsible  for  the  accuracy  of  the  MTADS  in  precisely  locating  and 
estimating  sizes  of  UXO  targets  and  in  detecting  deep  targets  in  complex  fields  with  nearby 
clustered  targets.  The  relatively  low  EM  coil  sensitivity  of  the  EMMS  system  used  in  the 
E-Range  surveys  led  to  missed  targets  and  relatively  poor  location  accuracy.  This  was  overcome 
by  the  redesigned  EMMS  system.  The  data  from  this  man- portable  unit  is  at  least  as  high  fidelity 
as  the  MTADS  vehicular  EM  survey  data. 

7.2.3  Target  Locations 

As  described  above,  precision  in  location  of  targets  is  a  strong  function  of  data  cleanup  during 
analysis.  However,  use  of  carefully  time- stamped  data,  the  highest  precision  GPS  navigation 
systems,  and  high  data  sampling  rates  are  equally  critical  in  obtaining  target  data  fits  with 
sufficient  information  to  precisely  describe  the  target.  In  general,  the  location  accuracy  of  the 
target  fitting  routines  from  a  vehicular  MTADS  magnetometer  survey  is  about  10-15  cm  whereas 
the  EM  array  typically  produces  fits  that  are  accurate  to  15-25  cm.  The  lower  accuracy  is,  in 
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part,  a  result  of  the  larger  coil  size  of  the  EM  array  and  the  lower  data  density  relative  to  the 
magnetometer  data.  The  accuracy  of  the  magnetometer  fitting  routines  in  predicting  target 
depths  is  typically  in  the  range  of  20-30%  of  the  absolute  depth.  Very  shallow  targets  often  have 
higher  errors  and  deep  targets  are  often  fit  to  within  10%  of  their  depth.  The  MTADS  baseline 
EM- fitting  routine  typically  gives  much  poorer  depth  fitting  accuracy  than  the  magnetometer¬ 
fitting  algorithm.  The  3-B  fitting  algorithm  applied  to  EM  data  typically  provides  depth 
estimates  that  are  more  accurate  than  the  baseline  point-dipole  EM  fit,  though  they  remain  less 
accurate  than  the  magnetometer  depth  predictions. 

Table  1 1  compares,  performance  predictions  of  the  vehicular  MTADS  magnetometer  array,  the 
MMS,  and  EMMS  surveys  with  the  JPG  ground  truth  (provided  by  Dr.  Ernesto  Cespedes  for 
Area  3).  To  provide  a  greater  range  of  information,  the  position  and  depth  predictions  of  the 
MMS  and  EMMS  surveys  were  compared  with  the  predictions  resulting  from  the  MTADS 
vehicular  survey.  The  latter  comparisons  might  reveal  any  systematic  bias  or  significant  random 
errors  in  the  location  accuracy  of  the  provided  ground  truth. 

Table  11.  Position  and  Depth  Discrepancies  in  the  Magnetometer  and  EM  Surveys 

Compared  with  JPG  Ground  Truth.* 


AX-Y  (m)  Relative 
to  JPG  Ground 
Truth 

A  Depth  (m) 
Relative  to  JPG 
Ground  Truth 

AX-Y  (m)  Relative 
to  MTADS  Mag 

ADepth(m) 
Relative  to 
MTADS  Mag 

Ave. 

Median 

Ave. 

Median 

Ave. 

Median 

Ave. 

Median 

EMMS 

0.18 

0.18 

0.16 

0.12 

0.21 

0.16 

0.21 

0.16 

MMS 

0.16 

0.11 

0.15 

0.11 

0.13 

0.09 

0.09 

0.05 

MTADS  Vehicle  Mag 

0.16 

0.09 

0.13 

0.08 

- 

- 

- 

- 

*  Both  UXO  and  clutter  targets  are  included  in  this  analysis. 


The  location  and  depth  predictions  of  the  vehicular  MTADS  survey  compare  favorably  with  the 
provided  ground  truth  and  fall  within  the  expected  accuracies  of  the  system.  There  was  no 
detectible  systematic  position  bias  nor  any  significant  single  target  location  discrepancy. 
Eikewise,  the  location  accuracies  and  depth  predictions  of  the  MMS  and  EMMS  surveys  and 
analyses  were  within  the  expected  accuracies  of  the  systems.  The  performance  of  the  MMS  and 
the  vehicular  MTADS  magnetometer  array  were  statistically  indistinguishable. 

7.2.4  Classification  Performance 

Differentiating  UXO  from  OE  scrap  and  metallic  clutter  has  been  the  primary  objective  of  every 
MTADS  development  program  for  the  past  3  years  (except  for  the  development  of  the  man- 
portable  MTADS  adjuncts). 

The  test  at  JPG  was  the  most  stringent  and  comprehensive  evaluation  of  the  latest  analysis  tool 
for  the  EM  system,  the  3-B  analysis  algorithm  [23].  The  approach  was  originally  developed  for 
the  MTADS  EM  array  with  the  overlapping  1-m  coil  systems.  It  was  adapted  for  this 
development  and  this  demonstration  with  the  single  unit  0.5- m  coil  system.  With  the  vehicular 
array,  two  orthogonal  surveys  of  the  same  area  are  typically  conducted  to  get  the  maximum 
possible  shape  information  for  the  processor.  In  the  present  instance,  the  smaller  coil  provides 
higher  resolution  data  and,  because  of  its  0.5  m^  total  area,  effectively  radiates  each  object  with  a 
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much  wider  range  of  incidence  radiation  vectors  on  each  survey  lane.  Using  the  EMMS,  data  are 
taken  on  closely  spaced  lanes  to  assure  overlap  and  high  data  density.  Under  these 
circumstances,  the  three  areas  were  surveyed  only  once,  and  the  processor  was  used  to  extract 
classification  information.  Classification  results  are  shown  in  Table  12.  Level  1  targets  were 
most  confidently  assigned  as  UXO.  Level  2  targets  are  less  confidently  assigned.  Level  3 
targets  are  doubtful,  but  are  more  likely  UXO  than  not.  Level  4  targets  are  declared  as  more 
likely  not  UXO.  Level  6  targets  are  most  likely  not  UXO,  and  it  was  recommended  that  those 
targets  not  be  dug.  Confidence  in  Level  5  targets,  although  not  likely  to  be  UXO,  was  not  high 
enough  to  risk  leaving  them  in  the  ground. 

Table  12.  Classification  of  the  Targets  from  the  JPG  Surveys.* 


Classification 
Confidence  Level 

AREA  3 

3-P  Analysis 

Joint  Analysis 

20-mm 

No  20-mm 

20-mm 

No  20-mm 

1 

22 

18 

20 

17 

2 

17 

6 

15 

9 

3 

14 

11 

25 

14 

4 

12 

8 

13 

8 

5 

28 

19 

42 

21 

6 

41 

72 

39 

85 

Total  declarations 

134 

134 

154 

154 

Dig/don’t  dig 

93/41 

62/72 

115/39 

69/85 

*Ground  truth  data  were  not  available  for  Area  1  and  2  when  this  document  was  completed. 


In  this  demonstration,  the  3-B  classifier’s  performance  was  a  limited  success.  The  pseudo-ROC 
curves  show  that  Area  3  was  the  easier  site  on  which  to  classify  targets.  Reaching  90%  correct 
UXO  declarations  on  Area  3  required  picking  only  about  60  non- UXO  targets.  Area  1  required 
more  than  80  targets  and  Area  2  more  than  90  targets  to  recover  90%  of  the  seeded  UXO. 

A  more  stringent  test  of  a  target  classifier  is  its  ability  to  recognize  OE  scrap  and  metal  clutter  as 
metallic  targets  that  can  be  left  in  the  field.  As  shown  in  Table  13,  using  the  3-B  classifier  at  the 
relatively  benign  Site  3,  11  of  29  (assuming  20- mm  projectiles  present)  or  21  of  34  (assuming 
20-mm  projectiles  not  present)  OE  scrap  targets  were  correctly  classified.  Only  4  of  29 
(assuming  20-mm  projectiles  present)  or  14  of  30  (assuming  20-mm  projectiles  not  present)  OE 
scrap  targets  in  the  field  were  correctly  not  recommended  for  digging.  In  the  ordnance  column, 
13  of  19  (assuming  20-mm  projectiles  present)  or  12  of  15  (assuming  20-mm  projectiles  not 
present)  anomalies  were  correctly  classified  as  UXO.  The  20-mm  projectiles  were  the  most 
difficult  UXO  targets  to  classify  correctly. 

Combining  the  EM  and  magnetometer  survey  data  in  a  joint  analysis  improved  both  the 
classification  and  dig  decisions  by  a  factor  of  about  two,  while  simultaneously  decreasing  the 
number  of  ordnance  items  that  would  be  left  in  the  field  after  digging.  The  correct  classification 
of  UXO  objects  in  the  joint  classification  was  similar  to  that  using  the  EM  data  alone  because 
these  classifications  were  still  primarily  based  on  the  3-B  analysis.  Most  improvement  in 
classification  in  the  joint  analysis  was  based  on  the  improved  spatial  resolution  in  the 
magnetometer  data  and  the  ability  of  the  magnetic  dipole  analysis  to  recognize  strong  remnant 
moments. 
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Table  13.  EMMS  Performance  Summary  for  Area  3, 


Targets 

Declared 

uxo 

Detected 

UXO  Not 
Detected 

Correct  UXO 
Classification 

UXO 

Misclassified 

UXO 

Correct 

Type 

Assignment 

Incorrect  UXO  Type 
Assignments 

OE  Scrap 
Correct 
Classification 

OE  Scrap 
Correctly 
Not  Dug 

1  3-3  classification  assuming  2(>-mm  projectiles  are  present  I 

134 

55 

20UXO 

35  OE 

19 

1  (81 -mm) 

13  of  19 

4  (20-mm) 

1  (60 -mm) 

1  (155-mm) 

7  of  13 

81mm  as  105mm/4.2in 
81mm  as  2.75in 

76mm  as  57/60mm 

105mm  as  76/8 1mm 
152mm  as  105mm/4.2in 
60mm  as  76/81mm 

11  of  29 

4  of  29 

1  3-3  classification  assuming  2l>-mm  projectiles  are  not  present  I 

134 

55 

16UX0 

39  OE 

15 

1  (60 -mm) 

12  of  15 

1  (60 -mm) 

1  (81 -mm) 

1  (155-mm) 

7  of  12 

81mm  as  105mm/4.2in 
81mm  as  2.75in 

76mm  as  57/60mm 

105mm  as  76/8 1mm 
152mm  as  105mm/4.2in 

21  of  34 

14  of  30 

1  1.1. 1.1.1  Joint  analysis  of  magnetometer  and  EM  surveys  assuming  20 -mm  projectiles  are  present  \ 

154 

55 

20UXO 

35  OE 

20 

0 

15  of  20 

4  (20-mm) 

1  (60 -mm) 

13  of  17 

81mm  as  57/60mm 

76mm  as  2.75in/81mm 
105mm  as  2.75in 

60m  as  76/8 1mm 

20  of  33 

10  of  33 

1  1.1. 1.1. 2  Joint  analysis  of  magnetometer  and  EM  surveys  assuming  20 -mm  projectiles  are  not  present  \ 

154 

55 

16UXO 

39  OE 

16 

0 

14  of  16 

1  (60 -mm) 

1  (81 -mm) 

12  of  16 

81mm  as  57/60mm 

76mm  as  2.75in/81mm 
105mm  as  2.75in 
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Where  time  and  resources  will  allow,  using  both  sensor  platforms  and  a  joint  target  analysis  will 
provide  improved  target  detection  and  recovery.  It  is  problematic  whether  it  will  lower  digging 
costs  overall.  The  improved  classification  capability  in  a  joint  analysis  tends  to  be  offset  by  the 
tendency  to  pick  more  targets  using  multiple  survey  data  sets. 
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8.0  THE  KAHO’OLAWE  ISLAND,  HAWAII,  DEMONSTRATION 
DESIGN 

ESTCP  funded  the  Naval  Explosive  Ordnance  Disposal  Technology  Division 
(NAVEODTECHDIV),  the  U.S.  Army  Engineer  Research  and  Development  Center  (ERDC), 
and  the  U.S.  Army  Environmental  Center  (AEC)  to  design  and  conduct  controlled  technology 
demonstrations  on  Kaho’olawe  Island  in  Hawaii  [7]. 

The  entire  island  was  used  for  many  years  as  an  air-to-ground  practice  range  for  a  wide  range  of 
ordnance.  At  the  time  of  this  demonstration,  an  active  clearance  had  been  underway  on  the 
island  for  some  time.  More  than  60,000  anomalies  had  been  dug  following  geophysical  UXO 
surveys  using  EM61  sensors.  On  average  >35  nonhazardous  objects  were  dug  for  each 
recovered  intact  UXO.  In  addition,  many  areas  were  resurveyed  a  second  (or  third)  time  with 
continued  recoveries  of  substantial  additional  ordnance.  Efficient  detection  of  UXO  is  difficult, 
primarily  because  the  volcanic  nature  of  the  soils,  the  presence  of  copious  amounts  of  shrapnel 
(of  all  sizes),  the  difficult  terrain,  limited  access  to  the  island,  and  the  constant  high  winds  further 
complicate  the  process.  The  ongoing  UXO  cleanup  operations  at  Kaho’olawe  have  been 
described  in  earlier  presentations  [22]. 

The  objective  of  these  demonstrations  was  to  evaluate  the  performance  of  a  new  generation  of 
EMI  sensor  technologies  for  potential  support  of  the  ongoing  cleanup  efforts  in  the  highly 
magnetic  noise  environment  [22].  Three  of  the  demonstrators  had  previously  used  the  same  or 
similar  systems  during  the  summer  and  fall  of  2000  to  conduct  UXO  surveys  at  the  JPG  on 
prepared  sites  containing  a  combination  of  UXO  and  OE  clutter  targets. 

Participating  in  the  Kaho’olawe  demonstration  were: 

•  NAEVA  (employing  a  new  Geonics  instrument,  the  EM63  on  a  wheeled  man-portable 
platform) 

•  GTE  (using  a  TM-5  EMU  frequency- domain  hand-held  sensor) 

•  Geophex,  Etd  (using  a  new  large  coil  version  of  their  GEM- 3  frequency  domain  sensor 
on  a  wheeled  man- portable  platform) 

•  NRE  (using  a  specially  designed  version  of  the  EM61  on  a  wheeled  man- portable 
platform) 

•  Parsons -UXB  (using  a  standard  wheeled  EM61,  as  deployed  on  the  island  for  QA 
studies) 

•  Parsons-UXB  (using  a  standard  wheeled  EM61  in  an  EM-and-flag  mode) 

Each  survey  group,  except  the  last,  used  an  integrated  GPS  to  create  geo- located  mapped  data 
files  for  target  reporting. 
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8.1  PERFORMANCE  OBJECTIVES 


The  project  managers  [24]  established  evaluation  objectives  for  these  demonstrations,  as  stated 
below: 


•  To  evaluate  the  demonstrators’  detection  and  discrimination  capabilities  by  means  of 
surveys  of  ten  30- m  X  30- m  grids  and  one  1-ha  area  within  the  Kaho’olawe  QA  range 
under  realistic  target/geologic,  clutter/man-made,  clutter/topography  scenarios  and  while 
operating  as  efficiently  as  possible  (minimizing  time,  manpower,  and  costs) 

•  To  evaluate  the  demonstrators’  ability  to  analyze  survey  data  in  a  timely  manner  and 
provide  prioritized  dig  hsts 

•  To  collect  manpower  and  time  data  required  to  produce  their  final  products  (prioritized 
dig  sheets  and  georeferenced  anomaly  maps) 

•  To  compare  the  performance  of  the  advanced  systems  with  the  baseline  technologies 
currently  employed  at  Kaho’olawe 

•  To  provide  high  quality,  ground -truthed,  georeferenced  data  for  post- demonstration 
analysis 

The  project  managers  established  the  following  factors  influencing  the  cost  and  performance 

evaluation  for  the  demonstrators: 

•  Equipment  setup  and  calibration  time  and  man-hour  requirements 

•  Time  and  man-hour  requirements  to  conduct  surveys 

•  Downtime  due  to  system  malfunctions  and  maintenance  requirements 

•  Re- acquisition/resurvey  time  and  man-hour  requirements 

•  Accuracy  of  georeferenced  maps  and  prbritized  dig  lists  with  respect  to 

Probability  of  detection  (Pd) 

False  alarm  rates  (Pfp,  FAR,  total  FAR) 

Discrimination  capability  (Pdisc) 

Identification  capability 
Target  location  accuracy 

8.2  SITE  CHARACTERISTICS 

The  levels  of  both  geological  (soil)  interference  and  OE  clutter  interference  are  much  higher  at 

Kaho’olawe  than  in  the  most  difficult  areas  of  the  JPG  survey.  At  the  time  these  demonstrations 

were  held,  about  1,100  acres  of  the  island  had  been  remediated.  Almost  all  of  the  geophysical 
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survey  work  had  been  done  using  classical  Geonics  EM61  units  operated  in  a  mag- and- flag 
mode.  Following  the  flagging  of  anomalies,  more  than  60,000  targets  were  dug.  More  than  37 
holes  have  been  dug  for  each  UXO  recovery.  False  alarms  attributed  to  metallic  objects  (mostly 
shrapnel)  are  about  three  times  more  prevalent  than  those  ascribed  to  geological  returns.  Many 
cleared  areas  have  been  resurveyed  and  dug  up  to  three  times,  recovering  more  intact  ordnance  in 
each  cycle.  The  overall  fraction  of  the  ordnance  removed  from  or  remaining  in  the  completed 
areas  remains  undetermined. 

Site  selection  criteria,  site  description,  and 
site  preparation  activities  are  described  in 
detail  in  the  Site  Preparation  Plan  [24],  but 
brief  descriptions  of  the  calibration  and  test 
sites  and  the  emplaced  targets  are  presented 
in  this  section.  Kaho’olawe  Island  was  used 
as  a  weapons  range  and  military  training 
area  from  1941  until  1990.  In  preparation 
for  the  Kaho’olawe  UXO  remediation 
project,  QC  and  QA  ranges  were 
established.  These  previously  cleaned  and 
seeded  areas  were  chosen  as  the  calibration 
and  demonstration  sites  for  this  project.  The 
QC  and  QA  areas  are  scoured  by  the 
continual  winds  that  are  typical  of  high 
elevations  on  the  island;  because  of  the 
elevation  and  winds,  no  significant 
permanent  soil  layer  has  been  created  by  sedimentation.  Significant  vegetation  has  been  missing 
from  this  part  of  the  island  for  more  than  60  years.  Precipitation  is  minimal,  and  vegetation  on 
the  QA  and  QC  sites  primarily  consists  of  scattered  small  clumps  of  desert  grass  that  collect 
blowing  sand.  There  are  isolated  1-2  foot  high  mounds  of  dirt,  outcroppings  of  hardpan  and/or 
lava,  and  occasional  large  rocks.  Fach  of  the  sites  has  shallow  gullies  and  depressions,  most  of 
which  can  be  traversed  by  the  wheeled  survey  platforms  with  some  bumping  and  bouncing. 
Figure  23. 

8.3  PHYSICAL  SETUP  AND  OPERATION 
8.3.1  Physical  Setup 

The  Calibration  Site.  The  calibration  area  consisted  of  three  30m  X  30m  grids  (see  Figure  24). 
The  north  and  south  grids  were  populated  with  previously  existing  UXO  and  frag  targets.  The 
FSTCP  grid  was  populated  with  UXO  and  frag  targets  specifically  for  this  project.  The  ground 
truth  of  all  items  emplaced  in  the  calibration  area  was  made  available  to  each  demonstrator  prior 
to  their  access  to  the  site.  The  seeded  items  are  graphically  indicated  in  Figure  24. 


Figure  23.  Calibration  Area  Within  the  QC  Range. 
Tapes  and  Sandbags  Were  Placed  by  the  Site 
Managers. 
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Figure  24.  Layout  of  the  Calibration  Site 
Showing  the  Locations  of  the  Calibration  Targets. 
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Figure  25.  Layout  of  the  QA  Demonstrati  on 
Area  Superimposed  on  the  30-Meter  Grid 
Structure. 


The  Demonstration  Site.  The  QA  Range  was  adapted  to  support  this  demonstration.  The 
layout,  shown  in  Figure  25,  is  based  on  a  set  of  30- m  x  30- m  grids.  Area  B  was  bounded  to 
encompass  an  area  of  1  hectare.  Areas  A  and  C  include  four  and  six  of  the  30- m  x  30- m  grids. 
Both  pre-existing  targets  and  frag  and  newly  buried  items  were  included  in  the  demonstration 
area.  Demonstrators  were  provided  with  UTM  coordinates  of  the  area  comers  and  first-order 
survey  control  points  near  each  site. 

Emplaced  Ordnance  and  OE  Clutter.  The  list  of  possible  inert  ordnance  items  for  burial  was 
provided  in  the  Demonstration  Test  Plan  [24].  All  ordnance  was  taken  either  from  previously 
fired  and  recovered  ordnance  on  Kaho’olawe  or  from  inert  stores  managed  by  AEC.  All 
ordnance  was  certified  as  inert;  unfired  ordnance  was  degaussed  prior  to  burial.  OE  clutter  items 
selected  from  scrap  and  shrapnel  recovered  on  Kaho’olawe  were  considered  typical  of  the 
ordnance  expended  on  the  site  [22].  UXO  challenge  items  included  air- fired  (20-40  mm) 
projectiles,  (2.25  in  and  2.75  in)  rockets,  (60-  and  81-mm)  ground- fired  mortars  and  (3-in  and  5- 
in)  projectiles  and  practice  bombs  (bomb  demonstration  unit  [BDU]-33  up  to  Mk-83). 
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8.3.2  Operations 


All  MTADS  operations  took  place  during  the 
weeks  of  15  and  23  October  2001.  Survey  data 
were  typically  taken  in  1-hour  increments, 
inspected  in  the  field  using  notebook  computers 
and  saved  to  hard  disk.  All  data  processing  and 
analysis  took  place  on  Maui  in  a  hotel  room  set 
up  as  an  office  (see  Figure  26).  Data  were 
downloaded  from  field  notebook  computers  to 
PCs.  One  PC  was  used  for  data  processing  and 
algorithm  development,  the  second  primarily  for 
target  analysis.  The  MTADS  survey  log  for  the 
operation  is  shown  in  Table  14.  The  first  week 
was  devoted  to  survey  work  on  the  calibration 
grids.  During  this  week  and  the  following 
weekend,  data  processing,  target  analysis,  and 
development  of  data  treatment  approaches  were  tested.  Actual  surveys  of  Areas  A,  B,  and  C 
took  place  during  October  22-24.  During  the  second  week  only  four  personnel  were  permitted  to 
travel  to  the  island;  each  day  one  person  remained  behind  on  Maui  to  process  and  analyze  data. 

Table  14.  MTADS  Kaho’olawe  Survey  Log. 


Figure  26.  Hotel  Room  on  Maui  Configured  as  a 
Data  Processing  Office. 


October  15- 16 

MMS  and  EMMS  surveys  of  the  three  30m  x  30m  calibration  grids  (6  data  files 
spanning  3.2  hrs  were  collected).  An  additional  short  test  was  performed  with  the 
EMMS  to  determine  reproducibility  of  data  by  walking  several  round  trips  of  the  same 
line. 

October  18 

Testing  of  the  EMMS  with  a  Tilt-meter  sensor  incorporated  for  more  accurate  sensor 
positioning  and  attitude  information.  (3  data  files  spanning  1  hour  were  collected). 
Implementation  of  the  Tilt-meter  sensor  ultimately  failed  due  to  unexpected  demands  on 
the  data  acquisition  system. 

October  22-23 

EMMS  survey  of  the  ten  30m  x  30m  grids,  areas  A  and  C  (11  data  files  spanning  4.78 
hrs  were  collected).  Note:  Several  breakdown  problems  were  encountered  with  the 
wheel-platform  attachment  assembly,  resulting  from  the  rough  terrain. 

October  23-24 

EMMS  survey  of  the  one  hectare  grid.  Area  B.  (13  data  files  spanning  4.64  hrs  were 
collected).  Note:  A  severe  breakdown  occurred  near  the  end  of  the  survey  that  could 
not  be  repaired.  As  a  result,  two  people  carried  the  sensor  platform  in  order  to  complete 
the  final  few  lines  of  the  survey. 

The  calibration  site  was  gridded  on  1.5-m  spacing  in  a  north/south  direction  using  twine  to  define 
the  survey  lanes.  Survey  tracks  were  0.5  m  apart,  and  80-d  steel  spikes  were  driven  at  1.5-m 
intervals  1  m  beyond  the  survey  boundary,  with  twine  stretched  between  spikes  to  define  the 
survey  grid.  The  site  was  surveyed  first  with  the  EM  system,  then  with  the  magnetometer 
platform. 

The  30- m  grids  and  the  1- hectare  areas  were  marked  as  a  single  site  in  a  northeast/ southwest 
direction  using  twine  on  1.5-m  spacing.  Survey  tracks  were  0.5  m  apart.  The  30- m  grids  were 
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completed  in  a  single  EM  survey  using  survey  tracks  stretching  the  entire  length  of  the  site.  The 
1 -hectare  site  was  completed  as  a  single  EM  survey,  beginning  at  the  northwest  boundary. 

8.4  SAMPLING  AND  MONITORING  PROCEDURES 

8.4.1  Production  Information 

Eigure  27  shows  a  course-over-ground  plot  for  the  survey  of  the  calibration  area.  This 
information  is  derived  from  the  GPS  data.  A  25- point  smooth  filter  has  been  applied  to  the  data, 
which  damps  some  of  the  side- to- side  rocking  motions  of  the  cart.  Similar  plots  are  shown  in 
Eigure  28  for  Area  B,  the  1-hectare  survey  and  in  Eigure  29  for  Areas  A  and  C.  As  is  apparent, 
the  calibration  area  and  Area  B,  the  1-hectare  survey,  are  relatively  smooth.  Effectively,  the 
entire  calibration  area  was  seamlessly  surveyed.  A  relatively  small  area  in  Area  C  near  the 
border  of  sections  E4  and  E5  was  missed  because  of  a  6- ft  deep  hole.  Areas  A  and  C  were 
considerably  more  rugged  with  two  gulleys  and  a  line  of  trees  stretching  along  the  long 
dimensions  of  the  survey.  These  obstacles,  along  with  brief  intermittent  navigation  dropouts 
(from  trees)  resulted  in  missed  survey  areas  amounting  to  a  small  percentage  of  the  targeted 
survey  area. 
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Figure  27.  Course-Over -Ground  Plot  for  the 
Calibration  Area  Survey. 


Figure  28.  Course-Over-Ground  Plot  for  the 
1-Hectare  Grid  Survey,  Area  B. 


Eigure  30  shows  a  survey  speed- over- ground  plot  for  a  20- minute  segment  of  the  1- hectare 
survey.  Twelve  consecutive  survey  lines  are  shown.  Eor  this  area,  the  average  survey  speed  was 
about  1.5  m/sec.  This  plot  is  also  derived  from  GPS  data. 
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Figure  29.  Course-Over-Ground  Plot  for  the  30- 
Meter  Grid  Survey,  Areas  A  and  C. 


Figure  30.  Plot  of  the  Survey  Speed  for  12 
Consecutive  Lanes  in  the  1 -Hectare  Survey. 


8.4.2  Data  Acquisition 

Figure  31  and  Figure  32  show  the  EMMS  and  MMS  surveying  on  the  ealibration  site.  The 
calibration  site  and  the  30- m  grids  and  1- hectare  grid  on  the  QA  range  were  surveyed  with  the 
EMMS;  only  the  calibration  site  was  also  surveyed  with  the  MMS. 


Figure  31.  The  EMMS  Surveying  on  the  Figure  32.  The  MMS  Surveying  on  the 

Calibration  Site.  Calihration  Site. 


8.5  ANALYTICAL  PROCEDURES 

The  EM  anomaly  map  for  the  calibration  site  is  shown  in  Eigure  33.  It  is  presented  as  an 
interpolated  image  on  a  175- mv  scale.  The  bright  features  stretching  along  the  west  and  east 
boundaries  are  returns  from  80-d  steel  spikes  that  were  used  to  stake  the  twine.  These  nails  also 
served  as  timing  fiduciary  markers  to  calibrate  the  timing  offsets  for  the  EM  system. 
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Figure  33.  EM  Survey  of  the  Kaho'olawe  Calibration  Grid  Using  a  20-Point  Demedian  Filter  in  an 
Interpolated  Image.  (The  seed  target  locations  are  marked  with  an  “x.”) 


Figure  34  shows  the  EM  anomaly  site  map  for  the  QA  site,  including  both  the  30-m  grids  and  the 
1 -hectare  area.  This  interpolated  image  is  presented  on  a  1,000- mv  scale.  Approximately  50  of 
the  brightest  targets  are  readily  apparent  on  this  scale,  as  are  the  80- d  steel  spikes  at  the  ends  of 
the  survey  lines. 

In  all  the  EM  presentations  in  this  document,  and  for  our  analysis,  the  EM  data  were  smoothed 
with  a  20- point  “down- the -track”  demedian  filter.  The  GPS  navigation  data  were  smoothed  with 
a  25 -point  filter,  and  the  EM  data  contains  a  timing  correction  of  3  ms.  The  demedian  filter 
suppressed  the  larger- scale  geological  features  without  affecting  the  presentation  or  fitting,  for 
most  UXO  targets.  During  the  EM  analysis,  a  data  set  was  also  displayed  that  was  smoothed 
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with  a  1,000- point  down- the- track  filter.  This  data  set  was  used  when  fitting  larger  UXO  targets 
whose  signatures  were  distorted  by  the  much  shorter  scale  20-point  filter. 
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Figure  34.  EM  Survey  of  the  QA  Site  at  Kaho'olawe  Using  a  20-Point  Demedian  Filter  in  an  Interpolated 
Image.  (Note  the  different  sensitivity  scale  from  Figure  33.) 
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9.0  KAHO’OLAWE  PERFORMANCE  ASSESSMENT 

9.1  THE  CALIBRATION  SITE 

Figure  35  shows  the  EM  survey  of  the  southwest  third  of  the  calibration  site,  both  as  an 
interpolated  image  and  as  a  pixel  plot.  Individual  data  points  are  sometimes  apparent  in  the  pixel 
plot.  The  overlay  of  the  target  locations  and  identities  shows  that  most,  but  not  all,  targets  are 
apparent  in  either  presentation.  In  the  calibration  site,  the  20-point  demedian  filter  effectively 
suppressed  much  of  the  interfering  geological  return.  Table  15  shows  a  summary  of  our  target 
analysis  of  the  calibration  site,  along  with  the  ground  truth  information  provided  by  ERDC.  The 
targets  in  the  ESTCP  (southwest  30- m  X  30- m)  g'id  are  much  more  easily  detected  than  the 
targets  on  the  remainder  of  the  calibration  site.  Overall,  in  the  calibration  site  we  concluded  that 
43  of  the  87  (-50%)  targets  would  likely  not  have  been  detected  without  prior  knowledge  of  the 
ground  truth.  The  targets  that  would  likely  be  missed  in  a  blind  EM  survey  are  highlighted  in 
yellow  in  the  right  column  in  Table  15.  The  smaller  targets,  up  to  and  including  many  of  the  60- 
and  81- mm  mortars,  would  not  be  detected  in  a  blind  survey,  primarily  because  of  geological 
interference.  Most  of  the  large,  deep  targets  (projectiles  or  bombs)  are  buried  below  the 
detection  limit  of  our  EMI  instrument.  The  deepest  targets  would  probably  be  beyond  our  EMI 
detection  limit,  even  without  the  geological  interference. 

Eigure  36  shows  a  presentation  on  a  similar  area  scale  of  an  anomaly  map  from  the 
magnetometer  survey.  The  geological  interference  is,  of  course,  much  worse  in  the 
magnetometry  data  than  in  the  EM  data.  The  data  shown  in  Eigure  36  is  highly  filtered  (15 
point,  down- the -track  demedian  filter).  Unfortunately,  we  did  not  have  access  to  more 
sophisticated  data  processing  filters  (or  the  time  to  incorporate  them)  to  improve  the 
magnetometer  analysis.  Many  of  the  targets  are  detectable  in  the  magnetometer  data;  however, 
only  the  deepest  large  targets  were  detectible  in  the  magnetometer  data,  but  not  in  the  EM  data. 
In  Eigure  36,  we  have  circled  dipole  signatures  that  are  probably  associated  with  the  identified 
target  assignments.  The  targets  circled  in  Eigure  36  were  declared  as  not  detectable  in  the  EM 
data  analysis.  Because  of  time  constraints  to  complete  the  surveys,  and  because  there  was 
insufficient  time  to  develop  new  data  processing  tools,  we  decided  not  to  conduct  magnetometry 
surveys  of  the  QA  site. 

9.2  THE  QA  DEMONSTRATION  SITE 

The  QA  site,  shown  in  Eigure  34,  is  much  more  highly  disturbed  than  the  calibration  site.  Eigure 
37  shows  a  pixel  presentation  of  part  of  the  IE  30- m  grid  on  about  the  same  scale  as  the  pixel 
plot  in  Eigure  35.  The  remainder  of  the  QA  site  looks  similar  when  viewed  on  this  presentation 
scale.  There  are  3  factors  affecting  the  QA  site  that  make  analysis  more  difficult  for  us  than 
analysis  of  the  calibration  site.  The  density  of  small  and  intermediate  sized  shrapnel  chunks  is 
much  higher  on  the  QA  site.  In  most  cases,  these  present  as  0.5-m  to  I- m  single-track  signals, 
such  as  those  that  dominate  the  image  in  Eigure  37.  The  surface  of  the  QA  site  is  also  physically 
much  rougher  than  the  calibration  site.  Our  EM  sensors,  unfortunately,  generate 
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Table  15.  MTADS  Target  Report  (edited),  Kaho’olawe  Calibration  Site,  20-Point  Smooth. 


MTADS 

ID 

Local  X 
(m) 

Local  Y 
(ml 

UTMX(m) 

UTM  Y  (m) 

Depth 

(m) 

Goodness 
of  Fit 

SIZE 

(m) 

Analyst  Comments 

RESULT 

1 

4.21 

8.43 

746574.24 

2273330.39 

0.73 

0.364 

0.032 

EO,  no  real  signal 

20MM,  Missed  TARGET 

2 

8.42 

6.57 

746578.45 

2273328.53 

0.67 

0.704 

0.058 

E6,  20mm,  better  pixel  target 

20MM,  OK 

3 

9.39 

10.85 

746579.42 

2273332.81 

0.75 

0.382 

0.046 

EM,  IPO  fit,  large  frag 

LARGE  FRAG,  OK 

4 

7.76 

12.59 

746577.79 

2273334.55 

0.72 

0.795 

0.086 

EK,  IPO  fit,  large  frag 

LARGE  FRAG,  OK 

5 

8.71 

14.94 

746578.74 

2273336.91 

0.90 

0.715 

0.097 

E7,  IPO  fit,  81mm 

81  MM,  OK 

6 

2.25 

18.03 

746572.28 

2273339.99 

0.72 

0.796 

0.049 

El ,  pixel  fit. 

20MM,  NOT  FIT  THE  TARGET 

7 

9.38 

20.13 

746579.41 

2273342.09 

0.55 

0.788 

0.063 

EG,  IPO  fit,  medium  frag 

MED  FRAG,  OK 

8 

5.11 

22.25 

746575.14 

2273344.22 

1.03 

0.463 

0.080 

EE,  IPO  fit,  medium  frag 

MED  FRAG,  OK 

9 

9.27 

24.15 

746579.30 

2273346.11 

0.74 

0.909 

0.088 

E8,  pixel  fit,  2.75in  WH 

2.75",  OK 

10 

16.13 

6.37 

746586.16 

2273328.34 

0.38 

0.888 

0.048 

EP,  IPO  fit,  medium  frag 

MED  FRAG,  OK 

11 

14.30 

12.74 

746584.33 

2273334.70 

0.59 

0.828 

0.070 

EL,  IPO  fit,  medium  frag 

MED  FRAG,  OK 

12 

17.27 

10.61 

746587.30 

2273332.58 

0.82 

0.672 

0.073 

E4,  IPO  fit,  40mm  pro! 

40MM,  OK 

13 

19.77 

14.98 

746589.81 

2273336.94 

0.64 

0.661 

0.091 

EJ  IPO  fit,  medium  frag 

MED  FRAG,  OK 

14 

16.62 

18.54 

746586.65 

2273340.50 

0.87 

0.704 

0.087 

E5,  IPO  fit,  105mm  HEAT 

105  HEAT,  OK 

15 

25.81 

21.15 

746595.84 

2273343.11 

0.72 

0.743 

0.053 

EH,  IPO  fit,  large  frag 

LARGE  FRAG,  OK 

16 

24.59 

14.72 

746594.62 

2273336.68 

1.23 

0.369 

0.096 

E2,  IPO  fit,  40mm  pro! 

40MM,  OK 

17 

25.42 

11.14 

746595.45 

2273333.11 

0.87 

0.467 

0.105 

EN,  pixel  fit,  no  real  signal 

SMALL  FRAG,  Mised  TARGET 

18 

31.93 

-0.86 

746601 .97 

2273321.11 

-2.38 

0.003 

0.414 

El,  pixel  fit,  no  real  signal 

20MM,  Missed  TARGET 

19 

18.09 

23.56 

746588.12 

2273345.53 

0.71 

0.657 

0.068 

EF,  IPO  fit,  medium  frag 

MED  FRAG,  OK 

20 

27.30 

25.55 

746597.33 

2273347.51 

7.93 

0.056 

1.939 

E3,  wont  fit  IPO,  60mm  mortar 

60MM,  Missed  TARGET 

21 

30.83 

23.28 

746600.87 

2273345.24 

0.33 

0.967 

0.067 

SO,  IPO  fit,  undefined  frag 

FRAG,  OK 

22 

32.92 

18.04 

746602.95 

2273340.00 

0.65 

0.772 

0.062 

2,  IPO  fit,  undefined  pipe 

PIPE,  OK 

23 

32.20 

13.79 

746602.23 

2273335.75 

0.49 

0.651 

0.057 

SA,  IPO  fit,  undefined  frag 

FRAG,  OK 

24 

31.71 

8.24 

746601 .75 

2273330.20 

0.34 

0.48 

0.036 

6,  IPO  fit,  60mm  mortar 

60MM  OK 

25 

29.40 

3.38 

746599.43 

2273325.34 

-1.83 

0.013 

0.489 

1 ,  IPO  wont  fit,  20mm,  no  signal 

20MM,  Missed  TARGET 

26 

35.65 

4.77 

746605.68 

2273326.74 

0.35 

0.592 

0.031 

7,  IPO  fit,  3in  proi 

3",  OK 

27 

32.50 

0.70 

746602.53 

2273322.67 

0.33 

0.595 

0.029 

18,  IPO  fit,  Mk81(5)1.8m,  weak  signal 

MK81,  FIT  GEOLOGY 

28 

42.72 

3.49 

746612.75 

2273325.45 

0.70 

0.485 

0.258 

1 7,  pixel  fit,  Mk81  @1 ,5m,  no  signal 

MK81,  FIT  GEOLOGY 

29 

40.37 

6.20 

746610.40 

2273328.16 

0.56 

0.665 

0.058 

12,  IPO  fit,  5in  proj  @1m 

5',  DOUBTFUL  FIT 

30 

52.18 

2.95 

746622.21 

2273324.91 

0.86 

0.892 

0.124 

16,  IPO  fit,  5001b  bomb  @1. 2m 

500LB,  OK 

31 

47.12 

5.83 

746617.15 

2273327.79 

0.34 

0.43 

0.029 

8,  IPO  fit,  3in  pipe 

3' PIPE,  DOUBTFUL  FIT 

32 

45.97 

7.62 

746616.00 

2273329.58 

0.60 

0.533 

0.053 

15,  IPO  fit,  3in  pipe 

3' PIPE,  DOUBTFUL  FIT 

33 

52.95 

7.18 

746622.98 

2273329.15 

0.76 

0.572 

0.086 

SE,  IPO  fit,  undefined  frag 

FRAG,  OK 

34 

52.29 

10.64 

746622.32 

2273332.60 

0.51 

0.756 

0.041 

23,  pixel  pick,  no  real  signal 

4  LB  BOMB,  Missed  TARGET 

35 

38.53 

14.51 

746608.56 

2273336.47 

0.40 

0.466 

0.133 

20,  wont  fit  IPO,  Mk82,  no  see 

MK82,  Missed  TARGET 

36 

45.80 

18.28 

746615.84 

2273340.25 

0.50 

0.757 

0.033 

5,  IPO  wont  fit,  5in  frag 

FRAG,  Missed  TARGET 

37 

40.49 

22.50 

746610.52 

2273344.47 

0.70 

0.555 

0.082 

14,  IPO  fit,  2.25in  rocket 

2.25"  OK 

38 

44.96 

22.49 

746614.99 

2273344.45 

0.60 

0.543 

0.058 

1 1 ,  IPO  fit,  2.25in  rocket 

2.25”,  Missed  TARGET 

39 

48.68 

23.53 

746618.71 

2273345.49 

0.84 

0.481 

0.077 

10,  IPO  fit,  81mm  mortar 

81  MM,  OK 

40 

56.22 

7.86 

746626.25 

2273329.82 

0.43 

0.845 

0.037 

9,  pixel  fit,  IPO  sees  geology,  2.25in 
rocket 

2.25”,  Missed  TARGET 

41 

60.90 

15.15 

746630.93 

2273337.12 

0.88 

0.28 

0.103 

4,  target  lost  in  border  nails 

5"  FRAG,  Missed  TARGET 

42 

58.67 

19.02 

746628.70 

2273340.99 

0.72 

0.565 

0.070 

SG,  IPO  fit,  undefined  frag 

FRAG,  Missed  TARGET 

43 

52.44 

17.01 

746622.47 

2273338.97 

0.56 

0.844 

0.057 

13  IPO  fit,  3in  pipe 

3",  OK 

44 

53.87 

17.30 

746623.90 

2273339.27 

0.29 

0.905 

0.045 

SL,  IPO  fit,  undefined  frag 

FRAG,  OK 

45 

55.75 

20.05 

746625.78 

2273342.01 

0.46 

0.784 

0.081 

SC,  IPO  fit,  undefined  frag 

FRAG,  OK 

46 

55.73 

21.95 

746625.76 

2273343.91 

0.67 

0.711 

0.052 

3,  pixel  pick,  no  see  in  IPO,  rocket 
motor 

rocket  motor,  MissedTARGET 

47 

53.43 

27.65 

746623.46 

2273349.62 

0.47 

0.433 

0.052 

19,  IPO  fit,  Mk  82  bomb,  doubtful 
signature 

MK82,  Missed  TARGET 

48 

52.91 

30.62 

746622.94 

2273352.59 

0.70 

0.556 

0.087 

SH,  IPO  fit,  undefined  frag 

FRAG,  OK 

49 

49.02 

26.46 

746619.06 

2273348.42 

0.42 

0.672 

0.050 

SO,  IPO  fit,  undefined  frag 

FRAG,  OK 

50 

42.99 

16.97 

746613.02 

2273338.93 

15.21 

0.001 

0.267 

21,  pixel  fit,  no  IPO  signal,  41b  bomb 

4LB,  MissedTARGET 

51 

46.90 

31.70 

746616.93 

2273353.66 

0.72 

0.523 

0.039 

20,  pixel  fit,  no  IPO  signal,  Mk82 

MK82,  Missed  TARGET 

52 

43.46 

33.08 

746613.49 

2273355.04 

0.67 

0.633 

0.078 

32,  IPO  fit,  40mm 

40MM,  OK 

53 

2544.08 

291.76 

749114.11 

2273613.72 

##### 

0.003 

21.402 

25,  pixel  pick,  no  signal,  5in  proj 

5",  MissedTARGET 

54 

44.25 

35.58 

746614.28 

2273357.54 

0.64 

0.769 

0.041 

31 ,  pixel  pick  no  IPO  signal,  40mm 

40MM,  Missed  TARGET 

55 

55.32 

34.54 

746625.36 

2273356.50 

0.78 

0.551 

0.065 

50  pixel  pick,  no  IPO  signal,  81mm  tail 

81  MM  TAIL,  Missed  TARGET 

56 

58.84 

36.54 

746628.87 

2273358.50 

0.85 

0.395 

0.085 

49,  IPO  fit,  81mm  tail 

81  MM  TAIL,  OK 

57 

53.26 

38.21 

746623.29 

2273360.17 

0.60 

0.73 

0.071 

48,  IPO  fit,  81mm  tail 

81  MM  TAIL,  OK 

58 

59.47 

41.79 

746629.50 

2273363.75 

0.66 

0.736 

0.052 

29,  IPO  fit,  60mm  mortar 

60MM,  OK 

59 

57.42 

43.60 

746627.45 

2273365.56 

0.85 

0.475 

0.080 

28,  pixel  fit,  IPO  no  fit,  60mm  mortat 

60MM,  Missed  TARGET 

56 


Table  15.  MTADS  Target  Report  (edited),  Kaho’olawe  Calibration  20-Point  Smooth. 

(continued) 


MTADS 

ID 

Local  X 
(m) 

Local  Y 
(m) 

UTMX(m) 

UTM  Y(m) 

Depth 

(m) 

Goodness 
of  Fit 

SIZE(m) 

Aqnalyst  Comments 

RESULT 

60 

60.69 

45.61 

746630.72 

2273367.57 

4.96 

0.077 

1.500 

27.  IPO  fit.  60mm  mortar 

60MM.  LOST  IN  GEOLOGY 

61 

49.27 

39.42 

746619.30 

2273361.38 

0.65 

0.554 

0.074 

SI,  IPO  fit,  undefined  fraq 

FRAG,  OK 

62 

********** 

*********** 

************* 

*************** 

********* 

0.021 

####### 

44,  IPO,  no  signal,  2.75ln  rocket 

2.75",  Missed  TARGET 

63 

44.52 

43.66 

746614.55 

2273365.62 

0.44 

0.705 

0.061 

43,  IPO  fit,  2.75ln  rocket 

2.75",  OK 

64 

43.04 

40.44 

746613.07 

2273362.41 

0.31 

0.37 

0.038 

47,  IPO  fit,  nose  fuze 

FUZE,  OK 

65 

39.62 

48.52 

746609.65 

2273370.48 

-2.43 

0.008 

0.756 

42,  pixel  pick  no  slqnal,  2.75ln  WFI 

2.75",  Missed  TARGET 

66 

37.44 

41.00 

746607.47 

2273362.96 

0.61 

0.515 

0.069 

SM,  IPO  fit,  undefined  frag 

FRAG,  OK 

67 

41.20 

55.67 

74661 1 .24 

2273377.64 

10.84 

0.009 

1.009 

24,  pixel  pick,  no  slqnal,  5ln  pro! 

5",  Missed  TARGET 

68 

33.43 

44.53 

746603.47 

2273366.49 

0.92 

0.029 

0.080 

46,  pixel  pick,  no  signal,  nose  fuze 

FUZE,  Missed  TARGET 

69 

32.24 

43.12 

746602.27 

2273365.08 

0.37 

0.074 

0.368 

53,  pixel  pick,  no  slqnal,  60mm  can 

60MM,  Missed  TARGET 

70 

21.37 

25.66 

746591 .40 

2273347.62 

0.56 

0.679 

0.070 

EC,  IPO  fit,  large  frag 

FRAG,  OK 

71 

14.93 

28.27 

746584.96 

2273350.23 

0.68 

0.509 

0.050 

EB,  IPO  fit,  medium  fraq 

FRAG,  OK 

72 

4.43 

27.30 

746574.46 

2273349.26 

0.72 

0.791 

0.095 

EA,  IPO  fit,  large  frag 

FRAG,  OK 

73 

35.32 

24.79 

746605.35 

2273346.75 

0.59 

0.577 

0.059 

1 8,  pixel  pick,  IPO  no  fit,  Mk81  (3)1 .8m 

MK81,  Missed  TARGET 

74 

33.94 

29.70 

746603.98 

2273351.66 

0.51 

0.776 

0.071 

SP,  IPO  fit,  undefined  frag 

FRAG,  OK 

75 

29.20 

28.98 

746599.23 

2273350.94 

0.67 

0.885 

0.066 

ED.  IPO  fit.  medium  fraq 

FRAG,  Missed  TARGET 

76 

58.64 

57.79 

746628.67 

2273379.76 

-4.62 

0.205 

2.176 

37.  pixel  pick,  no  IPO  slqnal,  81mm 

81  MM,  Missed  TARGET 

77 

53.53 

57.66 

746623.56 

2273379.62 

0.81 

0.714 

0.078 

36,  IPO  fit,  81mm  can 

81  MM,  Missed  TARGET 

78 

55.36 

52.95 

746625.39 

2273374.91 

0.68 

0.331 

0.046 

38.  pixel  Dick,  no  IPO  slqnal,  81mm 

81  MM,  Missed  TARGET 

79 

51.42 

50.34 

746621 .45 

2273372.30 

1.40 

0.521 

0.205 

SK,  IPO  fit,  undefined  fraq 

FRAG,  Missed  TARGET 

80 

********** 

*********** 

************* 

*************** 

********* 

NaN 

************ 

SJ,  pixel  pick,  no  signal,  undefined 
frag 

FRAG,  Missed  TARGET 

81 

49.35 

55.27 

746619.38 

2273377.24 

1.08 

0.389 

0.159 

SR,  pixel  geology  pick,  undefined  frag 

FRAG,  Missed  TARGET 

82 

41.47 

58.69 

74661 1 .50 

2273380.65 

0.85 

0.458 

0.068 

41 ,  IPO  fit,  2.25ln  rocket 

2.25",  Missed  TARGET 

83 

41.54 

54.49 

74661 1 .57 

2273376.45 

0.58 

0.371 

0.040 

40,  IPO  fit,  2.25ln  rocket 

2.25",  Missed  TARGET 

84 

39.12 

52.91 

746609.15 

2273374.88 

0.76 

0.439 

0.045 

39,  IPO,  no  slqnal,  2.25ln  rocket 

2.25",  Missed  TARGET 

85 

###### 

####### 

************* 

*************** 

********* 

0.011 

####### 

34,  IPO,  no  signal,  20mm 

20MM,  Missed  TARGET 

86 

35.64 

51.02 

746605.67 

2273372.98 

0.80 

0.73 

0.050 

33,  IPO,  no  slqnal,  20mm 

20MM,  Missed  TARGET 

87 

32.70 

57.28 

746602.73 

2273379.25 

0.50 

0.284 

0.048 

51,  IPO,  sees  qeoloqy,  60mm  can 

60MM,  Missed  TARGET 

*  The  designation  “IPO”  in  the  Comments  column  signals  that  the  target  data  were  selected  from  the  interpolated  image  presentation.  All  target 
fitting  algorithms  are  applied  to  the  uninterpolated  sensor  data  as  shown  in  the  pixel  plots 


sensor  spikes  when  the  system  bumps  hard  over  a  surface  feature.  These  signals,  which  one 
might  expect  to  be  sensor  data  spikes  that  could  be  edited  out,  unfortunately  damp  out  over  a 
period  of  a  few  tenths  of  a  second,  making  them  almost  undistinguishable  from  the  shrapnel 
clutter  signal  returns.  Together,  these  two  effects  are  responsible  for  most  of  the  red  signal 
return  shown  in  Figure  37.  This  noise- dominated  data  required  that  we  carry  out  analysis  at 
much  less  sensitive  scales  and  give  up  even  more  of  the  small  targets  than  in  the  calibration  site 
analysis.  The  third  effect,  referred  to  above,  is  the  more  intense  geological  interference  on  this 
site.  The  geological  returns  effectively  dictated  that  we  analyze  data  with  the  visual  guidance  of 
the  pixel  presentations  rather  than  the  combination  of  pixel  and  interpolated  image  presentations 
used  at  the  calibration  site. 
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Figure  35. 


Southwest  Quadrant  of  the  Calibration  Site.  A  Comparison  of  Pixel  Qeft)  and 
Interpolated  Image  Presentations  of  the  EM  Survey. 
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Figure  36.  Calibration  Site.  Interpolated  Image  from  the 
Magnetometry  Survey  Using  a  15-Point  Demedian  Filter. 
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The  EM  data  analysis  graphical  user  interface 
(GUI)  in  use  at  the  time  of  this  demonstration 
employed  what  we  call  the  3-B  analysis 
algorithm.  This  provides  an  analysis  that 
generates  three  parameters  that  presumably 
correlate  with  size  along  the  primary 
orthogonal  axes  of  the  target.  These  B 
parameters  are  used  as  clues  to  target  shape 
information  for  use  in  classification  decisions. 
The  geological  interferences  and  the  noise 
signals  from  the  shrapnel  clutter  and  sensor 
bouncing  rendered  the  3-B  parameters 
effectively  useless  as  an  analysis  tool  Figure 
38  shows  that  the  ordnance  on  the  calibration 
site  is  indistinguishable  from  clutter,  based  on 
the  3-B  analysis.  Therefore,  our  analysis  used 
the  previously  developed  baseline  EM  analysis 
approach  in  which  the  important  analysis 
parameters  are  location,  depth,  and  size. 
Shape  information,  to  the  extent  that  it  is 
available,  is  gleaned  from  the  detrend 
presentation  in  the  model  fit  analysis  window. 
The  goodness- of- fit  parameter  is  used 
primarily  as  an  evaluation  tool  to  guide  boxing 
the  data  chip  for  analysis  and  as  a  guide  in 
editing  the  data  selected  for  target  fitting. 


Figure  37.  Pixel  Presentation  of  Part  of  Grid  IE 
in  the  QA  Site  from  the  EM  Survey.  (The  pixel 
presentation  uses  the  same  settings  as  the  calibration 
site  presentation  in  Figure  35.) 
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Presentations  such  as  the  one  shown  in  Figure 
39  were  used  to  guide  the  analysis.  This 
presentation  shows  the  targets  boxed  for 
analysis  in  Grid  1C  of  Area  A.  Target 
numbers  were  deleted  so  the  sensor  data  could 
be  more  easily  visualized.  To  reduce  the 
signal  dominance  of  shrapnel  returns,  we  were 
guided  by  the  upper  coil  data.  We  particularly 
sought  out  targets  with  returns  in  adjacent 
tracks  on  the  assumption  that  larger  targets 
would  appear  on  multiple  tracks.  Some  single- 
track  targets  were  still  reported.  During 
analysis  of  individual  targets,  we  typically 
rescaled  the  presentation  many  times  to  seek 
out  smaller  targets  and  to  try  to  evaluate 

geological  returns.  However,  because  of  the  noisy  returns  we  were  limited  to  working  with 
fairly  high  intensity  signals.  We  were  aware  that  this  mitigated  against  our  detection  of  both  the 
small  targets  and  the  deep  targets  with  smaller  intensity  signal  returns. 
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Figure  38.  Three-Beta  Plot  from  the  Analysis  of 
the  Ordnance  and  Clutter  Targets  on  the 
Calihration  Site. 
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9.3  REPORTS 


Data  were  analyzed  from  the  hotel/office  site 
in  Maui,  and  the  required  target  tables  were 
submitted  to  the  program  representative.  On 
October  24,  preliminary  data  containing  a 
target  analysis  of  Area  A  were  submitted  to  the 
program  manager,  Scott  Steward.  The  analysis 
was  graded  immediately  (as  described  in  the 
Test  Plan)  [22],  and  results  were  returned  to 
NRL  to  guide  the  remaining  analysis.  The 
spreadsheet  in  Table  16  was  the  provided 
grading.  We  were  informed  that  we  had 
correctly  declared  -40%  of  the  seeded  targets 
in  our  preliminary  report.  Missed  items  were 
not  identified  and  further  ground  truth  was  not 
provided  until  the  final  report  [25].  The  list 
included  9  UXO  items  and  11  items  of  frag. 

These  results  were  consistent  with  our 
expectations  based  on  our  analysis  of 
calibration  site  data  and  the  relative  greater 
geological  and  shrapnel  interferences  on  the 

QA  site.  On  October  25,  we  delivered  to  Mr.  Steward  the  completed  analysis  and  the  remainder 
of  the  deliverables  specified  in  the  Demonstration  Test  Plan.  The  target  tables  were  submitted 
separately  for  the  1-hectare  site,  and  an  analysis  of  combined  Areas  A  and  C  was  submitted. 
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Figure  39.  QA  Site,  Grid  1C.  Pixel  Presentation 
of  the  EM  Survey  Showing  Targets  Selected  for 
Analysis. 
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Table  16.  Grading  from  Submission  of  Preliminary  Analysis  of  Targets  from  Area  A. 


UTM  Coordinates 

NRL  Tarqet  ID 

Northinq 

Easting 

Elev(msl) 

Description 

Depth  (m) 

Az  (deg) 

incl  (deg) 

Nose  U/D 

51 

2272256.419 

748397.026 

253.451 

Med  Fraa 

0.3 

90 

45 

49 

2272250.367 

748393.534 

252.521 

Large  Frag 

0.65 

135 

60 

41 

2272247.050 

748382.421 

252.332 

Large  Frao 

0.35 

0 

60 

56 

2272244.694 

748404.410 

252.861 

Med  Frag 

0.25 

315 

0 

50 

2272254.966 

748391.016 

252.964 

Med  Frag 

0.25 

270 

0 

65 

2272254.143 

748415.915 

253.774 

5"  HE  PRACTICE 

0.9144 

180 

0 

65 

2272254.728 

748415.651 

253.706 

82 

2272258.247 

748426.907 

254.370 

MK76  P.B.  (BDU  33-(NOSE) 

0.2286 

0 

0 

82 

2272258.102 

748426.634 

254.372 

1 

2272284.586 

748418.571 

255.406 

LAAW 

0.127 

0 

0 

1 

2272284.661 

748418.333 

255.404 

8 

2272271 .238 

748423.191 

254.656 

2.75"  ROCKET  WH 

0.381 

270 

45 

D 

8 

2272271 .667 

748423.305 

254.584 

14 

2272265.901 

748413.695 

253.946 

20  mm  Projectile 

0.25 

90 

0 

4,5 

2272277.843 

748421 .755 

254.518 

Large  Frao 

0.55 

0 

90 

13 

2272264.487 

748425.991 

254.555 

Med  Frao 

0.35 

90 

45 

18 

2272266.476 

748408.781 

253.920 

Small  Frag 

0.15 

45 

0 

228 

2272227.670 

748421 .220 

MK76  P.B.  (BDU  33) 

0.7112 

270 

45 

D 

226 

2272225.120 

748413.860 

MK-3  PRAC  BOMB 

0.2286 

0 

45 

D 

224,225 

2272220.170 

748407.530 

MK-81  P.  B.  W/S.E.  FINS 

1.016 

130 

0 

229 

2272223.595 

748423.136 

252.701 

Small  Frag 

0.1 

270 

0 

94 

2272258.720 

748439.050 

MK  76  P.B.  (BDU  33-(NOSE) 

0.2286 

0 

0 

231 

2272243.420 

748440.679 

254.520 

Med  Frao 

0.2 

0 

0 

232 

2272250.539 

748441 .520 

254.598 

Large  Frao 

0.55 

0 

0 
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At  the  ESTCP  Partners  Symposium  in  December  2001  a  preliminary  analysis  of  the  results  from 
all  demonstrators  for  the  1-hectare  site  was  presented  in  the  form  of  ROC  curves  showing 
relative  probability  of  UXO  detection  as  a  function  of  the  false  alarm  count  [26].  A  summary  of 
the  results,  presented  at  the  meeting,  is  shown  in  Figure  40.  NRL  was  Demonstrator  B.  The 
ESTCP  Program  Office  felt  that  NRE  should  have  picked  more  targets  in  our  analysis  because 
our  ROC  curve  (Demonstrator  B)  was  similar  in  shape  to  the  other  demonstrators,  but  our 
analysis  was  terminated  well  before  the  other  demonstrators  had  ceased  picking  targets  that 
continued  to  increase  their  performance  in  identifying  UXO  targets. 

It  was  further  agreed  between  the  ESTCP  Program  Office  and  the  Kaho’olawe  demonstration 
support  organizations  that  in  the  final  analysis  the  demonstrator’s  results  would  be  regraded 
following  deletion  of  the  20-  and  40-mm  targets  from  the  database.  This  presumed  that  the 
demonstrators  would  resubmit  analyses  based  on  this  assumption.  NRE  analyzed  and 
resubmitted  new  target  lists  for  evaluation.  In  the  reanalysis  a  few  small  targets  were  deleted  (or 
moved  to  lower  confidence  levels)  and,  as  requested,  many  additional  targets  were  chosen.  The 
total  targets  reported  for  Area  B  increased  from  141  to  305  over  the  analysis  submitted  in  Maui. 
Similarly,  the  final  count  of  targets  in  the  combined  Areas  A  -i-  C  increased  from  246  to  312. 

In  the  final  demonstration  report  [25],  the  ground  truth  for  all  sites  was  released  and  the 
performances  of  the  demonstrators  was  presented  and  analyzed.  Table  17  extracts  information 
from  this  report  relative  to  flie  NRE  performance  following  reanalysis  and  submission  of  the 
target  tables. 

Table  17.  Evaluation  of  the  NRL  Performance  from  the  Reanalyzed  Data  Assuming  the 

20-  and  40-mm  Ordnance  Were  Not  Present.* 


Area  A 

Area  B 

Area  C 

Total 

Number  of  Targets 
(without  20/40- mm) 

19 

55 

28 

102 

NRL  (without  20/40- mm) 

Targets  Detected 

8 

19 

8 

35 

Pd 

0.421 

0.345 

0.286 

0.343 

*  Adapted  from  Table  3  of  Reference  24 
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The  ROC  curves  showing  NRL’s  performance  in  each  of  the  areas  are  shown  in  Figure  41. 
These  data  reflect  the  deletion  of  the  20-  and  40-  mm  ordnance  from  the  database  and  are  based 
on  our  target  analyses  after  picking  additional  targets.  These  expanded  analyses  were  submitted 
to  ESTCP  and  ERDC  on  December  14,  2001.  The  information  in  Eigure  41  was  adapted  from 
Eigures  40-42  in  the  final  report  [25]. 
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Figure  40.  ROC  Curves  for  the  Five  Demonstrators’  Analyses  of  Area  A,  the  1 -Hectare  Site. 


9.4  RESULTS 

Based  on  the  Pd,  (Tables  2  and  3  of  Reference  25,  and  Eigures  40  and  41),  it  appears  that  in  Area 
B,  the  MTADS  ordnance  detection  efficiency  rose  substantially  as  the  result  of  the  reanalysis  of 
the  data,  i.e.,  from  21%  to  34.5%.  A  closer  inspection  of  all  the  information,  however,  shows 
that  almost  the  entire  increase  was  based  on  the  deletion  of  twenty- six  20- mm  and  40- mm  targets 
from  the  database.  In  our  original  submission  (on  Maui)  we  did  not  detect  any  of  the  20-  or  40- 
mm  targets  in  Area  B.  By  adding  164  targets  to  our  original  Area  B  dig  list  of  141  targets,  we 
detected  only  two  additional  targets  (a  60- mm  and  a  Mk-3).  Similarly,  by  expanding  the  number 
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Figure  41.  ROC  Curves  from  the  Reanalysis  and  Resubmission  of  NRL’s  Kaho'olawe  Data.  (Graphics  are  adapted  from  Figures  40-42  of  Reference  25.) 


of  target  picks  in  Areas  A  +  C  by  27%,  we  detected  only  three  additional  targets.  In  each  case,  at 
least  some  of  these  additional  “detected”  targets  may  have  been  coincidences.  On  the  basis  of 
either  the  detection  efficiency  or  ultimate  costs  to  remediate  targets  on  the  dig  list,  our  original 
target  analysis  as  submitted  on  Maui  was  the  better  of  the  two  work  products. 

The  important  questions  that  must  be  addressed,  however,  are: 

•  Why  was  the  MTADS  EMMS  performance  much  poorer  on  an  absolute  scale  on 
Kaho’olawe  than  had  been  previously  demonstrated  on  other  sites? 

•  Why  did  the  MTADS  EMMS  detect  a  much  smaller  fraction  of  the  Kaho’olawe  targets 
than  the  other  better  performing  demonstrators? 

Eirst,  consider  the  performance  of  the  three  systems  that  used  virtually  the  same  sensor — the 
MTADS  EMMS,  the  Parsons  EM-and-flag,  and  the  Parsons  EM  digital.  All  use  variants  of  the 
EM61.  The  EM61  has  excellent  sensitivity  for  very  small  shallow  ordnance,  good  detection 
efficiency  for  intermediate  sized  objects,  and  diminishing  detection  efficiency  for  more  deeply 
buried  objects.  The  ROC  curves,  Eigures  37-45  of  Inference  25,  show  that  the  two  Parsons 
EM61  approaches  performed  similarly;  overall,  the  EM  &  Elag  approach  may  have  been  the 
slightly  superior  of  the  two.  The  EMMS  ROC  curves  parallel  the  other  two  EM61  performance 
curves,  but  terminate  at  a  detection  limit  well  below  the  other  two  systems.  Picking  extra  targets 
from  the  EMMS  data  did  not  drive  the  performance  in  parallel  to  the  other  two  EM61  systems. 
One  of  the  historical  advantages  of  the  EMMS  (Eigure  31)  is  that  we  have  located  the  coils 
closer  to  the  ground  than  the  traditional  Geonics  wheeled  carts.  In  retrospect,  at  Kaho’olawe, 
this  was  a  disadvantage  because  survey  data  were  overwhelmed  by  returns  from  tiny  shrapnel 
fragments  (Eigure  37).  Eocating  the  sensors  higher  above  the  ground  would  provide  a  relative 
discrimination  against  this  very  small  clutter. 

A  substantial  impediment  to  detection  of  either  small  objects  or  weak  signals  by  the  EMMS  was 
the  overwhelming  noise  generated  by  bouncing  the  cart  over  rough  terrain.  Our  reanalysis  of  the 
data  demonstrated  that  there  were  effectively  no  further  gains  to  be  made  from  additional 
analysis  of  our  data.  Using  the  EM61  in  an  EM-and-flag  mode  allowed  the  cart  to  be 
maneuvered  back  and  forth  to  reacquire  signals  several  times  from  different  directions  and  while 
moving  at  different  speeds.  These  differences  provided  enough  of  a  detection  edge  that  the 
Parsons  EM61  systems  could  continue  successful  detection  further  up  the  ROC  curve  than  the 
EMMS.  Neither  of  the  Parsons  systems  could  detect  20- mm  ordnance;  however,  the  EM  &  Elag 
system  was  fairly  efficient  in  finding  the  40- mm  ordnance. 

It  is  likely  that  the  setup  of  the  control  electronics  on  the  EMMS  (higher  transmit  power,  earlier 
time  gates,  and  reduced  time  constants  imposed  during  the  data  acquisition  step)  may  also  have 
worked  to  a  relative  disadvantage  for  the  EMMS.  It  is  also  possible  that  the  experience  that 
Parsons  had  while  using  these  systems  specifically  on  Kaho’olawe  for  more  than  a  year  before 
this  demonstration  (in  conjunction  with  instant  feedback  from  digging  targets)  worked  to  the 
relative  advantage  for  their  system. 
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The  other  two  consistently  better  performing  systems  were  the  Geophex  GEM,  and  the  Geonics 
EM63,  operated  by  NAEVA.  The  EM63  performance  was  similar  to  that  of  the  two  Parsons 
EM61  systems,  except  in  Area  C  where  it  performed  relatively  less  well.  The  EM63  has  many 
time  gates.  It  is  not  clear  that  these  provided  any  significant  operational  advantage.  The  EM63 
also  has  relatively  lower  detection  sensitivity  than  the  EM61.  This  may  have  been  responsible, 
in  part,  for  the  relatively  poorer  performance  in  Area  C  where  the  EM63  failed  to  detect  seven  of 
the  eight  large/deep  bombs.  The  EM63  demonstrated  a  consistently  better  detection  capability 
than  the  EMMS.  Each  followed  similar  ROC  curves,  but,  apparently  better  signal- to- noise  levels 
allowed  the  EM63  to  ultimately  detect  -20%  more  ordnance  than  the  EMMS. 

The  Geophex  GEM  performed  outstandingly  well  in  Area  A  compared  to  all  other  systems  and 
about  equivalently  to  the  Parsons  EM61  surveys  in  Areas  B  and  C.  As  with  the  EM63,  the  GEM 
and  the  EMMS  initially  followed  generally  the  same  ROC  curve,  but  the  GEM  ultimately 
detected  10-20%  more  ordnance  than  the  EMMS.  See  Eigures  40-42  in  Reference  25. 

9.5  LESSONS  LEARNED 

The  relatively  poor  performance  of  the  EMMS  was  effectively  entirely  the  result  of  signal-to- 
noise  limitations  on  our  ability  to  analyze  data.  This  was  true  on  an  absolute  scale  relative  to  the 
EMMS  performance  at  other  sites  and  in  a  relative  sense  in  comparison  to  other  systems 
performing  at  Kaho’olawe.  There  are  three  primary  contributing  factors,  all  of  which  involve 
noise  contributions  (rather  than  limits  on  the  signal).  On  an  absolute  sensitivity  basis,  the 
EMMS  is  almost  certainly  the  most  sensitive  instrument  among  those  demonstrated  at 
Kaho’olawe. 

None  of  the  participants  demonstrated  the  ability  to  detect  20- mm  ordnance.  The  Parsons  EM- 
and-flag  (but  not  the  Parsons  digital  EM),  the  GEM,  and  the  EM63  detected  -half  of  the  40- mm 
ordnance.  Eor  the  EMMS  to  achieve  this  level  of  performance,  the  signals  generated  from 
shrapnel  and  from  cart  bouncing  would  ha’ve  to  be  substantially  reduced. 

Two  significant  alterations  would  have  to  be  taken  to  approach  this  goal.  Each  is,  to  a  large 
extent,  unique  to  this  site.  The  first  would  require  raising  the  coils  higher  above  the  ground, 
perhaps  even  higher  than  the  Parsons  systems.  Most  of  the  frag  pieces  are  very  small,  and  their 
signals  would  disappear  if  the  coils  were  raised.  The  second  “fix”  would  be  to  hand  carry  the 
EM  coils,  much  as  was  done  on  Adak,  Alaska,  because  of  the  rugged  terrain.  This  would  require 
two  operators  for  the  EMMS,  one  to  carry  the  coils  and  the  second  to  carry  the  support 
instrumentation  in  a  backpack.  Implementing  both  of  these  changes  might  improve  the  EMMS 
performance  to  the  level  of  (or  perhaps  marginally  better  than)  the  GEM,  the  EM63,  or  the  EM- 
and-flag  EM61.  It  seems  unlikely  that  any  fieldable  system  will  ever  be  able  to  detect 
significantly  more  than  50%  of  this  distribution  of  ordnance  on  this  site.  This  implies  that  it  is 
unlikely  that  even  three  repeated  surveys  and  digs  on  the  same  area  could  achieve  a  90% 
clearance. 

At  a  minimum,  it  is  now  well- documented  that  none  of  the  newer  technologies  demonstrated  in 
this  effort  offer  the  potential  for  substantial  improvements  over  the  use  of  a  standard  EM61  for 
detection  of  UXO.  This  conclusion  holds  only  for  this  site;  however,  this  site  is  very  unique. 
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Many  sites  are  rugged  and  difficult  to  survey,  and  many  sites,  to  a  greater  or  lesser  degree,  are 
characterized  by  magnetically  active  soils.  However,  this  site  has  magnetic  interferences  that  are 
as  difficult  as  any  naturally  occurring  soil  geology  anywhere  in  the  world.  The  site  is 
contaminated  by  UXO  widely  ranging  in  size  and  depths,  and  it  has  been  used  so  extensively  that 
the  surface  is  nearly  saturated  with  small,  medium,  and  large  shrapnel. 

The  Parsons  EM- and- flag  survey  that  was  done  as  part  of  this  demonstration  showed  a  similar 
performance  to  the  better  of  the  other  systems  demonstrated.  It  would  be  logical  to  conclude  that 
EM- and- flag  provides  an  efficient  and  economical  approach  for  cleaning  this  site.  Such  a 
conclusion  should  not  be  based  on  the  results  of  this  demonstration,  however.  The  performance 
in  a  Parsons  EM- and- flag  survey  is  completely  dependent  on  the  skill  of  and  consistent 
performance  by  the  equipment  operator.  Parsons,  no  doubt,  put  one  of  their  best  operators  on 
this  job.  His  performance  is  very  unlikely  to  be  the  same  as  that  of  the  dozens  of  other  operators, 
particularly  when  the  other  operators  work  very  long  days  under  very  difficult  conditions.  It  has 
been  generally  understood  for  many  years  in  the  UXO  community  that  the  performance  of  mag- 
and-flag  operators  is  highly  variable  from  operator  to  operator  and  that  achieving  consistent 
performance  by  a  single  operator  requires  that  he  work  continuously  only  for  relatively  short 
periods  of  time. 

It  is  much  more  feasible  to  achieve  a  consistent  UXO  survey  product  when  the  data  are  taken 
using  georeferenced  positioning,  routine  and  standardized  survey  data  collection  techniques,  and 
analysis  and  target  decisions  that  are  not  made  in  the  field  but  by  a  data  analyst  working  in  a 
controlled  environment. 

This  demonstration  took  place  under  very  realistic  conditions  on  an  extremely  challenging  site. 
The  developers  of  the  Technology  Demonstration  Plan,  the  site  preparation  effort,  the 
demonstration  support  staff,  and  the  performance  analysis  efforts  were  all  extremely  professional 
and  fair  to  all  participants.  The  fact  remains,  however,  that  the  demonstrators  brought  their  best 
equipment  and  their  best  people  and  that  the  operation  was  of  only  a  few  days  duration.  While 
the  results  demonstrate  how  well  each  of  the  techniques  could  perform  under  these  specific 
limited  circumstances,  they  do  not  directly  address  the  performance  any  of  these  technologies  in 
a  large-scale  clearance,  using  many  operators  over  an  extended  time  frame  such  as  that  taking 
place  on  Kaho’olawe.  The  results  are  likely  to  be  very  different. 

This  demonstration  did  prove,  however,  what  is  likely  the  maximum  achievable  result  for  the 
extended  operation  taking  place  on  Kaho’olawe.  It  should  be  concluded  that  one  cannot  expect 
to  consistently  achieve  an  efficiency  of  50%  UXO  recovery  on  this  site  in  a  single  pass  survey 
and  dig  operation.  The  QA  approach  that  is  being  used  on  the  island  takes  more  precisely- 
positioned  data  than  the  EM-and-flag  survey  but  is  likely  less  sensitive  because  of  signal-to- 
noise  limitations  from  taking  data  us  ing  a  cart  bumping  along  at  constant  speed.  The  results  of 
this  demonstration  should  be  accepted  as  the  ultimate  benchmark  of  maximum  expected 
performance  for  EM61  systems  in  this  environment. 
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10.0  COST  ASSESSMENT 


All  discussions  relating  to  cost  and  production  rates  are  based  on  the  JPG- V  demonstration  at  the 
Jefferson  Proving  Ground  as  described  in  Sections  6  and  7  of  this  document.  Careful  tracking  of 
onsite  costs  was  maintained  also  for  the  Kaho’olawe  demonstration  (Sections  8  and  9). 
However,  both  costs  and  production  rates  for  the  latter  demonstration  are  unrealistic  because  of 
the  transportation  costs  of  all  people  and  materials  by  air  to  Hawaii,  the  lodging  on  Maui  in 
resort  hotels,  the  difficult  and  limited  logistical  access  to  the  work  site,  and  the  2- week  extended 
operation  to  complete  effectively  only  3- hectares  of  survey  work. 

Operations  and  maintenance  costs  for  the  JPG-V  demonstration  include  labor  costs  associated 
with  setup,  calibration,  survey,  analysis,  and  maintenance  as  well  as  any  required  support 
equipment,  consumables,  and  supplies.  Labor  was  monitored  by  on-site  representatives  of  the 
government  and  converted  to  costs  by  the  application  of  the  same  labor  rates  for  all 
demonstrators.  The  following  labor  rates  were  used: 

•  Supervisor — $95/hour 

•  Data  Analyst — $57/hour 

•  Logistics/field  support — $28. 50/hour 

In  addition  to  the  costs  listed  above,  the  government  analyzed  the  dig  sheets  provided  by  each 
demonstrator,  and  cost  penalties  associated  with  false  positives  and  with  UXOs  misclassified  as 
clutter  were  assigned  as  described  in  Section  6.5. 

The  evaluation  factors  related  to  production  rate  performance  are  listed  in  Section  6.5  of  this 
report.  Because  of  a  variety  of  problems  experienced  in  the  field  by  some  of  the  demonstrators 
(e.g.,  the  need  to  collect  a  new  target  signature  library  on  site  and  the  inability  to  perform  on-site 
analysis),  direct  comparison  of  all  production  rate  factors  is  not  possible.  Because  survey  times 
are  important  production  factors,  and  accurate  information  from  all  the  demonstrators  (including 
mag- and- flag)  is  available,  it  was  decided  to  use  this  factor  for  production  rate  performance 
evaluation  and  comparison.  A  summary  of  the  time  and  man-hours  required  by  the  EMMS  to 
survey  each  site  is  presented  in  Table  18. 

Table  18.  EMMS  Production  and  Man-Hours. 


Area 

Number  of  People 

Time  On-Site  (Hr:Min) 

Actual  Man-Hours  (Hr:Min) 

1 

1-4 

5:49 

19:13 

2 

3-4 

7:55 

25:20 

3 

3 

5:56 

17:50 

The  standard  mag-and-flag  approach  achieved  an  average  production  rate  of  1  hectare  per  5.97 
hr  and  required  a  three-person  survey  crew.  The  best  performer  among  the  demonstrators  was 
the  EMMS,  which  achieved  an  average  of  1  hectare  per  6.56  hr  and  required  a  field  survey  crew 
ranging  from  one  to  four  persons. 
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The  costs  associated  with  each  field  task  are  detailed  in  Table  19  for  the  EMMS  and  summarized 
for  all  the  demonstrators  in  Table  20.  Field  costs  of  the  three  demonstrators  were  fairly  close, 
but  the  EMMS  demonstrated  the  lowest  costs.  The  baseline  mag- and- flag  field  work  conducted 
by  EODT  was  considerably  lower  than  all  three  demonstrators,  but  it  should  be  noted  that  EODT 
was  not  required  (nor  capable)  of  providing  georeferenced  sensor  maps,  prioritized  dig  lists,  and 
target  discrimination/classification.  Mag- and- flag  also  failed  to  reach  80%  Pd  at  any  of  the  three 
test  areas,  had  significantly  higher  false  alarms  at  the  high  magnetic  background  areas,  and  did 
not  meet  the  Kaho’olawe  clearance  requirements. 

Table  19.  Breakdown  of  Field  Costs. 


Area 

Categories 

Cost 

Time  (hr:  min) 

Job  Cost 

1 

Supervisor 

$95.00 

13:56 

$1,232.66 

Data  Analysis 

$57.00 

12:05 

$688.75 

Logistic/Field  Setup 

$28.50 

6:58 

$198.55 

Logistic/Field  Survey 

$28.50 

25:28 

$697.30 

Logistic/Field  Resurvey 

$28.50 

5:09 

$146.78 

Total 

$3,055.04 

2 

Supervisor 

$95.00 

18:32 

$1,760.66 

Data  Analysis 

$57.00 

26:16 

$1,497.20 

Logistic/Field  Setup 

$28.50 

11:22 

$323.95 

Logistic/Field  Survey 

$28.50 

25:20 

$722.00 

Logistic/Field  Resurvey 

$28.50 

0:00 

$0.00 

Total 

$4,303.81 

3 

Supervisor 

$95.00 

9:15 

$878.75 

Data  Analysis 

$57.00 

21:03 

$1,200.35 

Logistic/Field  Setup 

$28.50 

35:18 

$1,006.05 

Logistic/Field  Survey 

$28.50 

42:57 

$1,224.08 

Logistic/Field  Resurvey 

$28.50 

1:42 

$48.45 

Total 

$4,357.68 

Table  20.  Total  Cost  for  All  JPG-V  Test  Areas. 


Demonstrator 

Total  Cost  of  Field  Work 

Total  Cost  of  Field  Work 
Excluding  Data  Analysis 

NRL 

$11,183.24 

$7,854.44 

GEOPHEX 

$13,507.27 

$9,972.32 

NAEVA 

$10,940.68 

$10,783.93 

EODT 

$2,669.51 

$2,669.51 

Table  21  summarizes  the  operational  costs  of  the  demonstrator  systems  after  the  cost  penalties 
described  in  Section  6.5  were  applied.  These  penalties  consisted  of  $200  for  each  false  alarm 
(clutter  item  selected  for  digging  by  the  demonstrator)  and  the  cost  of  a  complete  resurvey  for 
one  or  more  UXO  targets  missed  or  erroneously  classified  as  clutter  with  high  confidence.  This 
table  highlights  the  fact  that  false  alarms  have  (by  a  large  margin)  the  greatest  impact  on  the  cost 
performance  of  each  system.  Table  21  indicates  that  all  three  demonstrators  were  penalized  with 
the  cost  of  a  resurvey  at  each  of  the  three  test  areas  because  UXO  had  been  left  in  the  ground  as  a 
result  of  misclassified  or  missed  targets.  The  EMMS  demonstrated  significantly  lower  overall 
costs  at  all  three  areas.  Comparison  with  the  baseline  mag-and-flag  costs  indicates  that  the  best 
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performing  EMI  technologies,  including  the  EMMS,  were  considerably  more  cost-effective. 
Even  though  EMI  system  costs  include  the  cost  of  analysis  in  both  the  survey  and  resurvey  cost 
factors,  they  are  consistently  lower  than  EODT’s.  As  expected,  the  EMI  advantage  is  more 
significant  in  Areas  I  and  2,  which  have  significant  levels  of  magnetic  noise  from  geologic 
sources. 

Table  21.  Demonstrator  Costs,  Including  Penalties  for  False  Alarms  and  Leaving  UXO 

Targets  in  the  Ground. 


Demonstrator 

Area 

Parameter 

NRL 

Geophex 

NAEVA 

EODT 

1 

Cost  of  survey 

$3,055 

$5,255 

$3,968 

$960 

Cost  of  resurvey 

$3,055 

$5,255 

$3,968 

$960 

Cost  of  false  alarms 

$14,200 

$18,200 

$15,000 

$26,900 

Total  cost 

$20,310 

$28,710 

$22,936 

$28,820 

2 

Cost  of  survey 

$4,304 

$5,094 

$2,247 

$912 

Cost  of  resurvey 

$4,304 

$5,094 

$2,247 

$912 

Cost  of  false  alarms 

$12,600 

$32,800 

$24,600 

$34,000 

Total  cost 

$21,208 

$42,988 

$29,094 

$35,824 

3 

Cost  of  survey 

$4,358 

$3,111 

$4,683 

$798 

Cost  of  resurvey 

$4,358 

$3,111 

$4,683 

$798 

Cost  of  false  alarms 

$13,000 

$31,000 

$15,800 

$20,600 

Total  cost 

$21,716 

$37,222 

$25,166 

$22,196 

Assuming  deployments  similar  to  those  at  JPG-V,  the  MTADS  deployment  costs  are  about 
$10,000  per  day,  on  site.  The  production  rate  of  the  MMS  and  EMMS  system  is  about  1.5- 
hectares  per  day  in  open  areas  using  GPS  navigation.  The  MMS  can  be  used  with  less  labor 
costs  as  it  is  much  less  labor-intensive  to  operate.  Production  rates  would  likely  decrease  and 
costs  would  likely  increase  by  a  factor  of  2  to  4  in  difficult  terrain  or  in  a  wooded  environment. 
Production  rates  with  the  vehicular  MTADS  system  are  7-10  hectares  per  day  on  areas  with 
terrain  typical  of  JPG.  Vehicular  production  efficiencies  are  higher  on  larger  sites  because  it  is 
possible  to  use  deployment  strategies  to  more  efficiently  survey  and  analyze  data.  Daily,  on-site, 
deployment  costs  are  similar  for  the  two  systems.  Mobilization  costs  are  typically  higher  for  the 
vehicular  system  because  it  requires  leasing  of  a  trailer  truck  and  fuel  and  labor  costs  associated 
with  transportation.  In  addition,  in  a  commercial  application,  amortization  or  depreciation  costs 
associated  with  the  vehicular  systems  will  be  significantly  higher  for  the  vehicular  system  than 
for  the  man- portable  adjuncts. 


Without  consideration  of  other  complicating  issues,  deployment  of  the  vehicular  system  to  sites 
where  it  can  be  used  is  probably  more  efficient  if  the  sites  are  larger  than  5  or  10  acres.  Eor  sites 
larger  than  20  acres,  it  would  be  difficult  to  rationalize  use  of  the  man-portable  systems  unless 
issues  of  availability  arise,  or  if  their  use  is  required  because  of  site  logistics.  Both  the  vehicular 
and  the  man-portable  systems  can  be  transported  in  the  same  trailer  and  could  likely  be  used 
simultaneously,  assuming  availability  of  the  GPS  equipment  and  the  required  labor  to  support 
simultaneous  operations. 
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